[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 431461109 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002354] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.007522] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.010166] LSM: Security Framework initializing [ 0.011048] Yama: becoming mindful. [ 0.012031] SELinux: Initializing. [ 0.014054] *** VALIDATE selinux *** [ 0.021292] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026523] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027169] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028091] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029121] *** VALIDATE tmpfs *** [ 0.030507] *** VALIDATE proc *** [ 0.032206] *** VALIDATE cgroup *** [ 0.033005] *** VALIDATE cgroup2 *** [ 0.034238] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035132] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037031] Spectre V2 : User space: Vulnerable [ 0.038005] Speculative Store Bypass: Vulnerable [ 0.041811] debug: unmapping init [mem 0xffffffffafe59000-0xffffffffafe60fff] [ 0.043185] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044657] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045022] ... version: 2 [ 0.046010] ... bit width: 48 [ 0.047007] ... generic registers: 4 [ 0.048008] ... value mask: 0000ffffffffffff [ 0.049009] ... max period: 00007fffffffffff [ 0.050009] ... fixed-purpose events: 3 [ 0.051007] ... event mask: 000000070000000f [ 0.053256] rcu: Hierarchical SRCU implementation. [ 0.055570] smp: Bringing up secondary CPUs ... [ 0.056573] x86: Booting SMP configuration: [ 0.057023] .... node #0, CPUs: #1 #2 #3 [ 0.061115] smp: Brought up 1 node, 4 CPUs [ 0.063016] smpboot: Max logical packages: 1 [ 0.064015] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.153553] node 0 deferred pages initialised in 88ms [ 0.157265] devtmpfs: initialized [ 0.158202] x86/mm: Memory block size: 128MB [ 0.161569] gcov: version magic: 0x41383552 [ 0.163308] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.167140] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.169434] pinctrl core: initialized pinctrl subsystem [ 0.171192] [ 0.171656] ************************************************************* [ 0.174014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.177015] ** ** [ 0.179011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.181013] ** ** [ 0.184013] ** This means that this kernel is built to expose internal ** [ 0.186013] ** IOMMU data structures, which may compromise security on ** [ 0.188013] ** your system. ** [ 0.190013] ** ** [ 0.193010] ** If you see this message and you are not debugging the ** [ 0.195010] ** kernel, report this immediately to your vendor! ** [ 0.197007] ** ** [ 0.199020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.201017] ************************************************************* [ 0.203795] NET: Registered protocol family 16 [ 0.205459] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.208051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.210071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.213143] cpuidle: using governor menu [ 0.214782] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.217124] PCI: Using configuration type 1 for base access [ 0.219112] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.227148] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.230063] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.234110] cryptd: max_cpu_qlen set to 1000 [ 0.239089] ACPI: Added _OSI(Module Device) [ 0.241022] ACPI: Added _OSI(Processor Device) [ 0.242011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.245021] ACPI: Added _OSI(Processor Aggregator Device) [ 0.248515] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.254472] ACPI: Interpreter enabled [ 0.255050] ACPI: PM: (supports S0 S3 S4 S5) [ 0.257018] ACPI: Using IOAPIC for interrupt routing [ 0.259178] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.262463] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.269000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.271043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.274023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.277119] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.284426] acpiphp: Slot [2] registered [ 0.285135] acpiphp: Slot [5] registered [ 0.287187] acpiphp: Slot [6] registered [ 0.288097] acpiphp: Slot [7] registered [ 0.290146] acpiphp: Slot [8] registered [ 0.293130] acpiphp: Slot [9] registered [ 0.295116] acpiphp: Slot [10] registered [ 0.296168] acpiphp: Slot [3] registered [ 0.298094] acpiphp: Slot [4] registered [ 0.299108] acpiphp: Slot [11] registered [ 0.300105] acpiphp: Slot [12] registered [ 0.302233] acpiphp: Slot [13] registered [ 0.304122] acpiphp: Slot [14] registered [ 0.305080] acpiphp: Slot [15] registered [ 0.307058] acpiphp: Slot [16] registered [ 0.308005] acpiphp: Slot [17] registered [ 0.309093] acpiphp: Slot [18] registered [ 0.311231] acpiphp: Slot [19] registered [ 0.312064] acpiphp: Slot [20] registered [ 0.313051] acpiphp: Slot [21] registered [ 0.314046] acpiphp: Slot [22] registered [ 0.315067] acpiphp: Slot [23] registered [ 0.316055] acpiphp: Slot [24] registered [ 0.317079] acpiphp: Slot [25] registered [ 0.319093] acpiphp: Slot [26] registered [ 0.320079] acpiphp: Slot [27] registered [ 0.321051] acpiphp: Slot [28] registered [ 0.323072] acpiphp: Slot [29] registered [ 0.324088] acpiphp: Slot [30] registered [ 0.325056] acpiphp: Slot [31] registered [ 0.326051] PCI host bridge to bus 0000:00 [ 0.328012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.329028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.332019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.333012] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.335017] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.338020] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.340180] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.343189] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.346780] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.355877] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.360074] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.363020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.365022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.369028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.372392] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.374599] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.377047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.379906] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.384016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.397756] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.402977] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.409461] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.416015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.422013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.439015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.451283] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.458017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.463020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.476025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.485940] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.490018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.496020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.510016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.518893] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.526026] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.531020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.547017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.558103] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.566013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.572014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.588018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.600321] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.608011] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.613011] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.628019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.640161] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.644475] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.647420] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.649287] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.651235] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.655086] iommu: Default domain type: Passthrough [ 0.658356] SCSI subsystem initialized [ 0.660092] ACPI: bus type USB registered [ 0.662081] usbcore: registered new interface driver usbfs [ 0.664050] usbcore: registered new interface driver hub [ 0.666063] usbcore: registered new device driver usb [ 0.667182] pps_core: LinuxPPS API ver. 1 registered [ 0.669013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.672043] PTP clock support registered [ 0.674069] EDAC MC: Ver: 3.0.0 [ 0.675463] PCI: Using ACPI for IRQ routing [ 0.676378] NetLabel: Initializing [ 0.677000] NetLabel: domain hash size = 128 [ 0.678009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.680073] NetLabel: unlabeled traffic allowed by default [ 0.682129] vgaarb: loaded [ 0.683242] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.685010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.689328] clocksource: Switched to clocksource kvm-clock [ 0.801821] VFS: Disk quotas dquot_6.6.0 [ 0.803164] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.805296] *** VALIDATE ramfs *** [ 0.806288] *** VALIDATE hugetlbfs *** [ 0.807649] pnp: PnP ACPI init [ 0.810016] pnp: PnP ACPI: found 6 devices [ 0.823979] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.826474] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.828300] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.830056] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.831984] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.833868] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.836355] NET: Registered protocol family 2 [ 0.840573] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.845066] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.849133] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.854699] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.858118] TCP: Hash tables configured (established 65536 bind 65536) [ 0.860985] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.863906] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.866561] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.869602] NET: Registered protocol family 1 [ 0.872824] RPC: Registered named UNIX socket transport module. [ 0.875308] RPC: Registered udp transport module. [ 0.877088] RPC: Registered tcp transport module. [ 0.878568] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.880943] NET: Registered protocol family 44 [ 0.882475] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.884341] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.886245] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.888217] PCI: CLS 0 bytes, default 64 [ 0.889703] Unpacking initramfs... [ 2.245407] debug: unmapping init [mem 0xffff8b03bcc54000-0xffff8b03bffbffff] [ 2.247958] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.249442] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.251473] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.733242] Initialise system trusted keyrings [ 2.734806] Key type blacklist registered [ 2.736295] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.745284] zbud: loaded [ 2.747929] *** VALIDATE nfs *** [ 2.748972] *** VALIDATE nfs4 *** [ 2.750357] pstore: using deflate compression [ 2.753820] Platform Keyring initialized [ 2.853499] NET: Registered protocol family 38 [ 2.855153] Key type asymmetric registered [ 2.856701] Asymmetric key parser 'x509' registered [ 2.858063] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.860605] io scheduler mq-deadline registered [ 2.862197] io scheduler kyber registered [ 2.863710] io scheduler bfq registered [ 2.866081] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.870625] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.873875] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.877117] ACPI: Power Button [PWRF] [ 2.882719] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.889669] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.904671] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.911542] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.925344] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.952737] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.982628] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.987682] Non-volatile memory driver v1.3 [ 2.989184] Linux agpgart interface v0.103 [ 3.014127] virtio_blk virtio1: [vda] 134816 512-byte logical blocks (69.0 MB/65.8 MiB) [ 3.016615] vda: detected capacity change from 0 to 69025792 [ 3.031122] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.033722] vdb: detected capacity change from 0 to 1073741824 [ 3.046219] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.049230] vdc: detected capacity change from 0 to 2621440000 [ 3.066072] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.068658] vdd: detected capacity change from 0 to 2621440000 [ 3.084599] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.087426] vde: detected capacity change from 0 to 4294967296 [ 3.100374] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.103076] vdf: detected capacity change from 0 to 4294967296 [ 3.109702] libphy: Fixed MDIO Bus: probed [ 3.114713] usbcore: registered new interface driver usbserial_generic [ 3.117068] usbserial: USB Serial support registered for generic [ 3.120120] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.124247] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.125939] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.128195] mousedev: PS/2 mouse device common for all mice [ 3.131822] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.134115] rtc_cmos 00:05: RTC can wake from S4 [ 3.138944] rtc_cmos 00:05: registered as rtc0 [ 3.142398] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.145422] intel_pstate: CPU model not supported [ 3.148559] hid: raw HID events driver (C) Jiri Kosina [ 3.150955] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.154701] usbcore: registered new interface driver usbhid [ 3.162626] usbhid: USB HID core driver [ 3.163887] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.165395] drop_monitor: Initializing network drop monitor service [ 3.170409] Initializing XFRM netlink socket [ 3.175147] NET: Registered protocol family 10 [ 3.178548] Segment Routing with IPv6 [ 3.180421] NET: Registered protocol family 17 [ 3.182646] mpls_gso: MPLS GSO support [ 3.190252] RAS: Correctable Errors collector initialized. [ 3.192791] AVX version of gcm_enc/dec engaged. [ 3.194586] AES CTR mode by8 optimization enabled [ 3.297953] sched_clock: Marking stable (3297804106, 0)->(4120241974, -822437868) [ 3.302508] registered taskstats version 1 [ 3.304456] Loading compiled-in X.509 certificates [ 3.307069] zswap: loaded using pool lzo/zbud [ 3.332926] Key type big_key registered [ 3.360631] Key type encrypted registered [ 3.362711] ima: No TPM chip found, activating TPM-bypass! [ 3.365244] ima: Allocated hash algorithm: sha1 [ 3.367481] ima: No architecture policies found [ 3.369860] evm: Initialising EVM extended attributes: [ 3.372226] evm: security.selinux [ 3.373869] evm: security.ima [ 3.375240] evm: security.capability [ 3.376888] evm: HMAC attrs: 0x1 [ 3.382324] rtc_cmos 00:05: setting system clock to 2026-04-30 23:50:17 UTC (1777593017) [ 3.389501] debug: unmapping init [mem 0xffffffffb0e03000-0xffffffffb0ffffff] [ 3.393299] debug: unmapping init [mem 0xffffffffafb82000-0xffffffffafe58fff] [ 3.401112] Write protecting the kernel read-only data: 28672k [ 3.405929] debug: unmapping init [mem 0xffffffffae203000-0xffffffffae3fffff] [ 3.408932] debug: unmapping init [mem 0xffffffffaeb14000-0xffffffffaebfffff] [ 3.449886] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.458565] systemd[1]: Detected virtualization kvm. [ 3.460482] systemd[1]: Detected architecture x86-64. [ 3.462510] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.488643] systemd[1]: No hostname configured. [ 3.491032] systemd[1]: Set hostname to . [ 3.492864] random: systemd: uninitialized urandom read (16 bytes read) [ 3.495424] systemd[1]: Initializing machine ID from random generator. [ 3.677278] random: systemd: uninitialized urandom read (16 bytes read) [ 3.681878] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.685218] random: systemd: uninitialized urandom read (16 bytes read) [ 3.687627] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.695109] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.482460] device-mapper: uevent: version 1.0.3 [ 4.485461] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 5.261733] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.310715] virtio_net virtio0 ens2: renamed from eth0 [ 5.456122] scsi host0: ata_piix [ 5.476374] scsi host1: ata_piix [ 5.478317] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.481473] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.889293] random: crng init done [ 9.893105] random: 7 urandom warning(s) missed due to ratelimiting [ 9.922414] dracut-initqueue[580]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.515309] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.307848] printk: systemd: 26 output lines suppressed due to ratelimiting [ 13.839808] SELinux: Disabled at runtime. [ 13.985454] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.996363] systemd[1]: Detected virtualization kvm. [ 13.998153] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 15.532608] systemd[1]: initrd-switch-root.service: Succeeded. [ 15.539662] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 15.552984] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 15.569310] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 15.580881] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 15.691080] systemd[1]: Starting Journal Service... Starting Journal Service... [ 15.867590] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK [0[ 16.314276] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS m] Reached target Slices. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 16.950482] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 17.725862] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 17.844080] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 18.126143] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 18.187718] EDAC sbridge: Ver: 1.1.2 [ 21.698575] Key type dns_resolver registered [ 22.077473] NFS: Registering the id_resolver key type [ 22.080192] Key type id_resolver registered [ 22.082321] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg425-server login: [ 60.911815] libcfs: loading out-of-tree module taints kernel. [ 60.930032] Key type ._llcrypt registered [ 60.931374] Key type .llcrypt registered [ 60.974583] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 69.379965] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 70.069910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 70.081333] alg: No test for adler32 (adler32-zlib) [ 71.115660] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 71.451501] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 73.079143] Key type lgssc registered [ 73.736578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 80.816655] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 97.833815] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 104.045160] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 104.058512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 105.189617] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 105.208913] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 105.257709] Lustre: lustre-MDT0000: new disk, initializing [ 105.309446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 105.320733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 107.088639] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 113.291185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 113.342426] Lustre: 6471:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 113.365859] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 113.370157] Lustre: Skipped 1 previous similar message [ 113.415478] Lustre: lustre-MDT0001: new disk, initializing [ 113.451930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 113.467763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 113.474812] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 115.128973] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 117.804657] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 122.333385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 122.467056] Lustre: lustre-OST0000: new disk, initializing [ 122.469981] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 122.503492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 122.961495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 122.967355] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 123.010793] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 125.318663] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 132.496314] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 132.573412] Lustre: lustre-OST0001: new disk, initializing [ 132.576266] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 132.600283] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 135.522936] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 140.279889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 140.283996] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 140.312030] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 142.793464] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 146.820654] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 149.653935] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing check_logdir /tmp/testlogs/ [ 151.705557] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing yml_node [ 153.664454] Lustre: DEBUG MARKER: Client: 2.17.52.126 [ 155.094042] Lustre: DEBUG MARKER: MDS: 2.17.52.126 [ 156.343648] Lustre: DEBUG MARKER: OSS: 2.17.52.126 [ 157.126611] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Apr 30 19:52:50 EDT 2026 [ 164.820270] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 165.544790] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 180.192051] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 180.195478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 180.204753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 181.216443] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 181.216618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 181.223812] Lustre: Skipped 1 previous similar message [ 181.226948] Lustre: Skipped 1 previous similar message [ 184.119135] Lustre: server umount lustre-MDT0000 complete [ 185.916997] LustreError: 6463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777593200 with bad export cookie 9953380731585620360 [ 185.918507] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 185.923031] LustreError: 6463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 186.337523] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 186.337815] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 186.347444] Lustre: Skipped 2 previous similar messages [ 190.367738] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 190.370660] Lustre: Skipped 2 previous similar messages [ 192.127976] Lustre: server umount lustre-MDT0001 complete [ 193.810650] Lustre: server umount lustre-OST0000 complete [ 195.398133] Lustre: server umount lustre-OST0001 complete [ 197.697501] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 200.912041] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 215.682446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 215.802888] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 215.820898] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 215.864180] Lustre: lustre-MDT0000: new disk, initializing [ 215.913910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 215.927717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 217.658545] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 223.440893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 223.511698] Lustre: 14932:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 223.683204] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 225.601311] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 229.508469] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 231.217320] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 234.172425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 234.331733] Lustre: lustre-OST0000: new disk, initializing [ 234.334383] Lustre: Skipped 1 previous similar message [ 234.336491] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 234.338356] Lustre: Skipped 2 previous similar messages [ 234.381309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 234.384980] Lustre: Skipped 1 previous similar message [ 235.927155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 235.931849] Lustre: Skipped 1 previous similar message [ 235.936605] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 235.993455] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 237.117440] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 240.805888] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 242.616247] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 242.727532] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 244.663936] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 244.786235] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 246.245477] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 246.245860] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 246.249183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 246.253364] Lustre: Skipped 1 previous similar message [ 251.833698] Lustre: server umount lustre-OST0000 complete [ 254.435715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 254.439237] LustreError: Skipped 1 previous similar message [ 254.441953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 254.448991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 254.455428] Lustre: Skipped 3 previous similar messages [ 268.255236] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 268.340648] Lustre: server umount lustre-MDT0000 complete [ 269.942224] LustreError: 14923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777593284 with bad export cookie 9953380731585621879 [ 269.943035] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 269.946981] LustreError: 14923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 270.092637] Lustre: server umount lustre-MDT0001 complete [ 271.812845] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 19:54:45 (1777593285) [ 272.818640] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 273.554407] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 19:54:47 (1777593287) [ 276.359953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 276.580317] LustreError: 18931:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 276.589886] LustreError: 18931:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 276.617256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 278.314661] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 281.976094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 282.079843] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 282.082964] Lustre: Skipped 6 previous similar messages [ 283.642426] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 285.830782] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 287.259538] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 290.330062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 293.228600] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 295.597478] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 312.290703] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 312.291242] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 312.296243] Lustre: Skipped 2 previous similar messages [ 315.427883] Lustre: server umount lustre-OST0000 complete [ 317.409093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 317.411827] Lustre: Skipped 3 previous similar messages [ 331.231121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 331.296995] Lustre: server umount lustre-MDT0000 complete [ 332.760336] LustreError: 18910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777593346 with bad export cookie 9953380731585622761 [ 332.762844] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 332.765263] LustreError: 18910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 332.895257] Lustre: server umount lustre-MDT0001 complete [ 337.060412] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 338.325470] Key type lgssc unregistered [ 338.475472] LNet: 22127:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.479464] LNetError: 22127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 338.489751] LNet: Removed LNI 192.168.204.125@tcp [ 338.903351] Key type .llcrypt unregistered [ 338.905216] Key type ._llcrypt unregistered [ 349.049470] Key type ._llcrypt registered [ 349.051082] Key type .llcrypt registered [ 349.098944] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 349.507480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 349.564582] alg: No test for adler32 (adler32-zlib) [ 350.430295] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 350.529247] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 352.119231] Key type lgssc registered [ 352.623401] Lustre: Echo OBD driver; http://www.lustre.org/ [ 356.881797] loop: module loaded [ 356.915319] loop0: detected capacity change from 0 to 204800000 [ 357.079605] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 357.090810] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 357.280148] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 360.051718] loop0: detected capacity change from 0 to 204800000 [ 360.126577] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 360.132218] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 360.140606] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 362.517121] loop0: detected capacity change from 0 to 204800000 [ 364.976843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 364.983980] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 366.090473] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 366.108744] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 366.148406] Lustre: lustre-MDT0000: new disk, initializing [ 366.177513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 366.188079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 367.625923] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 371.998343] loop1: detected capacity change from 0 to 204800000 [ 374.604911] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 374.733540] Lustre: lustre-OST07c6: new disk, initializing [ 374.736379] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 374.740724] Lustre: Skipped 1 previous similar message [ 374.779241] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 375.620319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 375.625838] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 375.641474] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 377.263438] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 379.718121] Lustre: 24259:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 379.722752] Lustre: 24259:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 379.730560] Lustre: 24259:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 379.735246] Lustre: 24259:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 379.740087] Lustre: 24259:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 379.745332] Lustre: 24259:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 394.330049] Lustre: 24260:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 394.333916] Lustre: 24260:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 394.337611] Lustre: 24260:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 394.341314] Lustre: 24260:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 394.345043] Lustre: 24260:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 394.348091] Lustre: 24260:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 397.914895] Lustre: 24881:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 397.918696] Lustre: 24881:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 397.921851] Lustre: 24881:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 397.924545] Lustre: 24881:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 397.927686] Lustre: 24881:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 397.931475] Lustre: 24881:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 397.935067] Lustre: 24881:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 397.938426] Lustre: 24881:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 397.941707] Lustre: 24881:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 397.945318] Lustre: 24881:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 397.948618] Lustre: 24881:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 397.952318] Lustre: 24881:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 401.032895] Lustre: 24259:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 401.039504] Lustre: 24259:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 401.044888] Lustre: 24259:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 401.050323] Lustre: 24259:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 401.055354] Lustre: 24259:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 401.060135] Lustre: 24259:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 401.065096] Lustre: 24259:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 401.069700] Lustre: 24259:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 401.073418] Lustre: 24259:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 401.076523] Lustre: 24259:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 401.084335] Lustre: 24259:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 401.091268] Lustre: 24259:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 407.770033] Lustre: 24260:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 407.774739] Lustre: 24260:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 407.777555] Lustre: 24260:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 407.780875] Lustre: 24260:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 407.784636] Lustre: 24260:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 407.789950] Lustre: 24260:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 407.794294] Lustre: 24260:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 407.798079] Lustre: 24260:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 407.801319] Lustre: 24260:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 407.804431] Lustre: 24260:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 407.807157] Lustre: 24260:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 407.810311] Lustre: 24260:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 418.006534] Lustre: 24881:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 418.011396] Lustre: 24881:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 418.015386] Lustre: 24881:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 418.018089] Lustre: 24881:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 418.021882] Lustre: 24881:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 418.025056] Lustre: 24881:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 418.028538] Lustre: 24881:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 418.032186] Lustre: 24881:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 418.036027] Lustre: 24881:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 418.039147] Lustre: 24881:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 418.043205] Lustre: 24881:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 418.046715] Lustre: 24881:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 462.816943] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 462.820759] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 462.827227] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 465.233127] Lustre: server umount lustre-OST07c6 complete [ 468.653572] Lustre: server umount lustre-MDT0000 complete [ 474.776550] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 475.937487] Key type lgssc unregistered [ 476.075347] LNet: 27775:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 476.079271] LNetError: 27775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 476.089679] LNet: Removed LNI 192.168.204.125@tcp [ 476.427473] Key type .llcrypt unregistered [ 476.429358] Key type ._llcrypt unregistered [ 484.677248] Key type ._llcrypt registered [ 484.678853] Key type .llcrypt registered [ 484.733250] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 485.150073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 485.193947] alg: No test for adler32 (adler32-zlib) [ 486.069520] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 486.158373] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 487.751196] Key type lgssc registered [ 488.196144] Lustre: Echo OBD driver; http://www.lustre.org/ [ 491.246114] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 491.259221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 492.444096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 493.907515] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 497.082899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 497.219058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 498.634349] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 500.667432] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 502.116654] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 504.675946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 504.808532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 506.931986] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 508.968346] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 518.124343] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 527.841949] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 527.842032] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 527.845478] LustreError: Skipped 1 previous similar message [ 527.851498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 528.355236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 528.355777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 528.356544] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 528.365388] Lustre: Skipped 1 previous similar message [ 531.155369] Lustre: server umount lustre-OST0000 complete [ 532.959988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 532.963250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 532.969823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 538.592095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 538.594370] Lustre: Skipped 3 previous similar messages [ 538.865687] Lustre: server umount lustre-MDT0000 complete [ 540.372504] LustreError: 30622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777593554 with bad export cookie 7851228828168393349 [ 540.375091] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 540.377838] LustreError: 30622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 540.515095] Lustre: server umount lustre-MDT0001 complete [ 544.708190] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 545.864929] Key type lgssc unregistered [ 546.005488] LNet: 32404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 546.008741] LNetError: 32404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 546.016625] LNet: Removed LNI 192.168.204.125@tcp [ 546.363511] Key type .llcrypt unregistered [ 546.365117] Key type ._llcrypt unregistered [ 554.383658] Key type ._llcrypt registered [ 554.385343] Key type .llcrypt registered [ 554.432057] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 554.897188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 554.906507] alg: No test for adler32 (adler32-zlib) [ 555.804843] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 555.903429] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 557.495218] Key type lgssc registered [ 557.909412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 560.875436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 560.884559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 562.088906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 563.628891] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 567.052486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 567.202822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 568.703834] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 570.839791] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 572.266897] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 575.149231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 575.286503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 577.558614] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 579.679944] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 587.564574] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 597.984527] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 597.987955] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 597.995518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 597.999025] Lustre: Skipped 1 previous similar message [ 601.236754] Lustre: server umount lustre-OST0000 complete [ 603.105434] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 603.105638] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 603.106259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 603.110937] Lustre: Skipped 2 previous similar messages [ 603.114751] LustreError: Skipped 1 previous similar message [ 608.224518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 608.226766] Lustre: Skipped 2 previous similar messages [ 608.790969] Lustre: server umount lustre-MDT0000 complete [ 610.161270] LustreError: 36093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777593624 with bad export cookie 9615797309810373777 [ 610.164577] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 610.165532] LustreError: 36093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 610.277291] Lustre: server umount lustre-MDT0001 complete [ 614.171860] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 615.303568] Key type lgssc unregistered [ 615.448586] LNet: 36827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 615.452444] LNetError: 36827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 615.464465] LNet: Removed LNI 192.168.204.125@tcp [ 615.805666] Key type .llcrypt unregistered [ 615.807595] Key type ._llcrypt unregistered [ 625.813222] Key type ._llcrypt registered [ 625.815040] Key type .llcrypt registered [ 625.868316] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 626.214670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 626.230596] alg: No test for adler32 (adler32-zlib) [ 627.130015] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 627.228940] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 628.839179] Key type lgssc registered [ 629.330751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 632.925403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 632.935501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 634.141859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 635.740315] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 639.119464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 639.314440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 640.974987] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 643.157781] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 644.453536] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 646.752251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 646.881362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 648.929286] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 650.842228] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 659.174823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 664.547902] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 664.548625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 664.552993] Lustre: Skipped 2 previous similar messages [ 664.556236] Lustre: Skipped 1 previous similar message [ 666.013927] Lustre: server umount lustre-MDT0000 complete [ 666.512821] LustreError: 38229:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 666.519718] LustreError: 38229:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 667.372327] LustreError: 39819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777593681 with bad export cookie 12925892074274018389 [ 667.374417] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 667.376731] LustreError: 39819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 669.664245] LustreError: 40155:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 669.664462] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 669.676149] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 671.625189] LustreError: 38200:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 671.634980] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.25@tcp (stopping) [ 673.511662] Lustre: server umount lustre-MDT0001 complete [ 691.405349] Lustre: server umount lustre-OST0000 complete [ 696.861102] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 697.898955] Key type lgssc unregistered [ 698.024420] LNet: 41739:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 698.027212] LNetError: 41739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 698.034542] LNet: Removed LNI 192.168.204.125@tcp [ 698.328869] Key type .llcrypt unregistered [ 698.330175] Key type ._llcrypt unregistered [ 707.326639] Key type ._llcrypt registered [ 707.327627] Key type .llcrypt registered [ 707.362787] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 707.666503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 707.688047] alg: No test for adler32 (adler32-zlib) [ 708.542755] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 708.631055] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 710.215189] Key type lgssc registered [ 710.717946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 713.581703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 713.589659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 714.770335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 716.303832] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 719.254398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 719.386355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 720.911222] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 722.924590] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 724.246784] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 726.634789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 726.754809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 728.876898] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 730.723533] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 740.072668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 745.440945] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 745.441244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 745.444060] LustreError: Skipped 1 previous similar message [ 745.449752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 745.453704] Lustre: Skipped 1 previous similar message [ 746.383546] Lustre: lustre-OST0000: Not available for connect from 192.168.204.25@tcp (stopping) [ 749.583647] Lustre: server umount lustre-OST0000 complete [ 762.191341] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 762.195729] Lustre: Skipped 1 previous similar message [ 762.198521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 762.201643] Lustre: Skipped 1 previous similar message [ 765.280285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 765.282823] Lustre: Skipped 1 previous similar message [ 768.342831] Lustre: server umount lustre-MDT0000 complete [ 769.766098] LustreError: 43095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777593783 with bad export cookie 14935715464860736367 [ 769.769731] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 769.770277] LustreError: 43095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 784.351193] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 784.436489] Lustre: server umount lustre-MDT0001 complete [ 788.091140] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 789.128209] Key type lgssc unregistered [ 789.249431] LNet: 46489:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 789.252678] LNetError: 46489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 789.261287] LNet: Removed LNI 192.168.204.125@tcp [ 789.546651] Key type .llcrypt unregistered [ 789.548049] Key type ._llcrypt unregistered [ 798.368550] Key type ._llcrypt registered [ 798.369770] Key type .llcrypt registered [ 798.406372] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 798.714616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 798.756463] alg: No test for adler32 (adler32-zlib) [ 799.629402] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 799.723145] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 801.311093] Key type lgssc registered [ 801.636378] Lustre: Echo OBD driver; http://www.lustre.org/ [ 804.072719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 804.083019] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 805.229898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 806.484044] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 809.182441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 809.302695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 810.533297] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 812.328945] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 813.425962] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 815.639791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 815.754091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 817.497718] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 819.225092] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 830.121387] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 837.053483] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 838.151802] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 848.899822] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-04-3020h04m22s [ 849.435148] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 850.318463] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.25@tcp (stopping) [ 850.399855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 850.402346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 851.335214] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.25@tcp (stopping) [ 851.339039] Lustre: Skipped 3 previous similar messages [ 856.342580] Lustre: server umount lustre-MDT0000 complete [ 857.472500] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 857.478516] LustreError: 50025:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 857.483747] LustreError: 50025:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 859.696463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 859.748107] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 859.852444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 859.874616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 860.901291] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 861.225991] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 862.444735] Lustre: DEBUG MARKER: Wait for df (36003) ... [ 863.021248] Lustre: DEBUG MARKER: done [ 865.248523] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 865.252457] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 865.257625] Lustre: Skipped 2 previous similar messages [ 865.260417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 865.263028] Lustre: Skipped 3 previous similar messages [ 866.193048] Lustre: server umount lustre-OST0000 complete [ 869.668305] Lustre: server umount lustre-MDT0000 complete [ 871.096555] LustreError: 47844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777593885 with bad export cookie 10191337565350278419 [ 871.100781] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 871.102716] LustreError: 47844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 881.686751] Lustre: server umount lustre-MDT0001 complete [ 885.035547] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 886.039803] Key type lgssc unregistered [ 886.161326] LNet: 52428:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 886.163801] LNetError: 52428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 886.172297] LNet: Removed LNI 192.168.204.125@tcp [ 886.460105] Key type .llcrypt unregistered [ 886.461550] Key type ._llcrypt unregistered [ 899.945850] Key type ._llcrypt registered [ 899.947285] Key type .llcrypt registered [ 899.984736] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 900.325937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 900.377697] alg: No test for adler32 (adler32-zlib) [ 901.229143] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 901.320154] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 902.903106] Key type lgssc registered [ 903.305160] Lustre: Echo OBD driver; http://www.lustre.org/ [ 905.952210] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 905.964619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 907.086556] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 907.097677] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 907.167064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 908.595369] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 911.568661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 911.599917] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 911.611264] Lustre: 54594:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 911.707114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 913.090662] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 914.921384] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 916.146436] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 918.263348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 918.330383] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 918.406816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 919.894074] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 920.359705] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 922.133156] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 925.461280] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-04-3020h05m39s [ 925.996616] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 928.398911] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 929.190227] Lustre: DEBUG MARKER: Creating a test file and stat it [ 929.203536] LustreError: 54603:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 949.658899] Lustre: lustre-MDT0000: Client c939ab69-c8a1-493c-a7dc-1954c5c03171 (at 192.168.204.25@tcp) reconnecting [ 969.247125] LustreError: 54603:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 970.013703] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 970.901808] Lustre: DEBUG MARKER: done [ 974.817367] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 974.820726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 974.826100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 976.354485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 976.355396] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 976.362089] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 978.647661] Lustre: server umount lustre-OST0000 complete [ 981.472370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 981.473121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 981.476839] Lustre: Skipped 1 previous similar message [ 981.481245] Lustre: Skipped 2 previous similar messages [ 987.552283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 987.554683] Lustre: Skipped 3 previous similar messages [ 993.760178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 993.762303] Lustre: Skipped 1 previous similar message [ 994.271101] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 994.330331] Lustre: server umount lustre-MDT0000 complete [ 995.488995] LustreError: 54589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777594009 with bad export cookie 16376179196799199122 [ 995.491697] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 995.493202] LustreError: 54589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 995.601566] Lustre: server umount lustre-MDT0001 complete [ 998.869157] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 999.855310] Key type lgssc unregistered [ 999.980377] LNet: 58565:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 999.982895] LNetError: 58565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 999.992259] LNet: Removed LNI 192.168.204.125@tcp [ 1000.277421] Key type .llcrypt unregistered [ 1000.278616] Key type ._llcrypt unregistered [ 1015.865773] Key type ._llcrypt registered [ 1015.867224] Key type .llcrypt registered [ 1015.902758] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 1016.211567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1016.253436] alg: No test for adler32 (adler32-zlib) [ 1017.096429] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 1017.175027] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 1018.751155] Key type lgssc registered [ 1019.105119] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1020.176564] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1020.183594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1021.281681] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1021.291677] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1021.357797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1021.722595] Lustre: Failing over lustre-MDT0000 [ 1021.828794] Lustre: server umount lustre-MDT0000 complete [ 1023.857681] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 20:07:17 (1777594037) [ 1026.171887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1026.318001] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1027.583439] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1030.455965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1030.489373] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1030.499851] Lustre: 61322:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1030.596343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1031.812250] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1033.488398] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1034.636759] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1036.835075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1036.898039] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1036.951207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1038.769337] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1040.552085] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1046.034300] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1054.062422] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1056.226020] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1056.227062] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1056.228096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1056.229842] Lustre: Skipped 1 previous similar message [ 1061.344184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1061.346638] Lustre: Skipped 4 previous similar messages [ 1061.712448] Lustre: server umount lustre-MDT0000 complete [ 1062.890197] LustreError: 61312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777594077 with bad export cookie 5640728268240023479 [ 1062.892399] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1062.893616] LustreError: 61312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1063.009229] Lustre: server umount lustre-MDT0001 complete [ 1064.083811] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1065.548869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1068.247314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1073.631873] LustreError: 60244:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d34d6937 x1863942438682624/t0(0) o250->MGC192.168.204.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1073.766907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1073.799227] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1075.038205] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1077.785829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1077.874374] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1077.880243] Lustre: Skipped 1 previous similar message [ 1079.194871] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1080.945510] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1082.163595] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1082.847143] Lustre: 60246:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777594080/real 1777594080] req@00000000ac6392a4 x1863942438682240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777594096 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1083.363184] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1086.439646] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1087.455087] Lustre: 60248:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777594085/real 1777594085] req@00000000d9b1fcac x1863942438682496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777594101 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1087.465554] Lustre: 60248:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1089.951176] Lustre: 60245:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777594087/real 1777594087] req@000000001fe6eb15 x1863942438682752/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777594103 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1091.553018] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1091.554395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1091.557131] Lustre: Skipped 2 previous similar messages [ 1091.561653] Lustre: Skipped 2 previous similar messages [ 1095.058371] Lustre: server umount lustre-MDT0000 complete [ 1096.371048] LustreError: 63576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777594110 with bad export cookie 5640728268240025747 [ 1096.374287] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1096.376365] LustreError: 63576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1096.485969] Lustre: server umount lustre-MDT0001 complete [ 1098.615427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1101.749082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1105.887632] LustreError: 60244:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000090109e44 x1863942438702336/t0(0) o250->MGC192.168.204.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1105.998335] LustreError: 62941:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1106.003882] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1106.005725] LustreError: 62941:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1106.020054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1106.021921] Lustre: Skipped 1 previous similar message [ 1106.036277] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1107.233215] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1109.857552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1111.014861] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1111.168713] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1112.031068] Lustre: 60246:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777594110/real 1777594110] req@000000002510cae8 x1863942438702080/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777594126 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1112.812209] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1113.983502] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1116.129148] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1116.130085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1116.134025] Lustre: Skipped 1 previous similar message [ 1117.151148] Lustre: 60247:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777594115/real 1777594115] req@00000000f0c466d5 x1863942438702208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777594131 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1121.248624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1121.250456] Lustre: Skipped 4 previous similar messages [ 1121.748401] Lustre: server umount lustre-MDT0000 complete [ 1122.900031] LustreError: 61312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777594137 with bad export cookie 5640728268240027126 [ 1122.902766] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1122.904396] LustreError: 61312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1123.008605] Lustre: server umount lustre-MDT0001 complete [ 1125.041716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1126.067080] Lustre: DEBUG MARKER: files compared the same [ 1136.932288] Lustre: server umount lustre-OST0000 complete [ 1142.223808] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 1143.150316] Key type lgssc unregistered [ 1143.264336] LNet: 70248:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1143.267389] LNetError: 70248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1143.274226] LNet: Removed LNI 192.168.204.125@tcp [ 1143.541172] Key type .llcrypt unregistered [ 1143.542190] Key type ._llcrypt unregistered [ 1152.069509] Key type ._llcrypt registered [ 1152.070867] Key type .llcrypt registered [ 1152.108380] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 1152.471932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1152.498734] alg: No test for adler32 (adler32-zlib) [ 1153.365030] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 1153.452874] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 1155.031182] Key type lgssc registered [ 1155.357603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1157.841075] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1157.846927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1159.004461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1160.201375] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug +malloc all [ 1162.868449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1162.957170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1164.210299] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug +malloc all [ 1165.898095] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1166.980794] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1169.191552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1169.277502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1170.339781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1171.074140] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug +malloc all [ 1172.767091] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1178.593921] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1178.594334] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1178.594721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1178.598969] Lustre: Skipped 1 previous similar message [ 1182.349877] Lustre: server umount lustre-OST0000 complete [ 1183.711668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1183.711722] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1183.711892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1183.711896] Lustre: Skipped 1 previous similar message [ 1183.716132] Lustre: Skipped 1 previous similar message [ 1188.831650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1188.834647] Lustre: Skipped 2 previous similar messages [ 1189.710397] Lustre: server umount lustre-MDT0000 complete [ 1190.904611] LustreError: 71605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777594205 with bad export cookie 13874481456025819521 [ 1190.906553] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1190.908601] LustreError: 71605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1191.018345] Lustre: server umount lustre-MDT0001 complete [ 1194.478614] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 1195.521359] Key type lgssc unregistered [ 1195.646326] LNet: 74796:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1195.648491] LNetError: 74796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1195.657279] LNet: Removed LNI 192.168.204.125@tcp [ 1195.914548] Key type .llcrypt unregistered [ 1195.916290] Key type ._llcrypt unregistered [ 1204.576642] Key type ._llcrypt registered [ 1204.577942] Key type .llcrypt registered [ 1204.614210] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 1204.909927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1204.936110] alg: No test for adler32 (adler32-zlib) [ 1205.783084] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 1205.861514] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 1207.439120] Key type lgssc registered [ 1207.783081] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1210.296142] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1210.302733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1225.759391] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1230.879185] LustreError: 76145:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.125@tcp: failed processing log, type 4: rc = -110 [ 1257.503163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1259.350200] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1261.032296] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1263.536388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1263.581301] Lustre: *** cfs_fail_loc=706, val=0*** [ 1263.703102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1263.729511] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1264.985943] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1267.986994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1268.116445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1269.325103] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1271.137128] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1272.380569] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1273.312184] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1273.315532] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1273.320608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1273.824216] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1273.827475] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1273.832504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1275.146874] Lustre: server umount lustre-OST0000 complete [ 1277.408517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1277.409293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1277.414270] Lustre: Skipped 1 previous similar message [ 1282.528219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1282.530782] Lustre: Skipped 2 previous similar messages [ 1282.577828] Lustre: server umount lustre-MDT0000 complete [ 1283.862050] LustreError: 76151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777594297 with bad export cookie 16140018594949035729 [ 1283.865459] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1283.866625] LustreError: 76151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1283.971779] Lustre: server umount lustre-MDT0001 complete [ 1287.320396] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 1288.362956] Key type lgssc unregistered [ 1288.482347] LNet: 79400:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1288.485219] LNetError: 79400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1288.496265] LNet: Removed LNI 192.168.204.125@tcp [ 1288.758625] Key type .llcrypt unregistered [ 1288.759873] Key type ._llcrypt unregistered [ 1297.749256] Key type ._llcrypt registered [ 1297.750613] Key type .llcrypt registered [ 1297.793927] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 1298.258888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1298.292192] alg: No test for adler32 (adler32-zlib) [ 1299.159472] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 1299.245255] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 1300.831177] Key type lgssc registered [ 1301.207922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1303.699365] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1303.706325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1305.976605] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1308.419135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1308.582575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1309.795718] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1312.338131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1312.446626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1314.132193] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1316.723284] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1316.727085] Lustre: MGS: Client 5154cc8c-7ccc-4366-b008-f9f13898a64a (at 0@lo) reconnecting [ 1316.733052] LustreError: 81458:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1316.800456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1316.833706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1317.993246] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1327.072386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1327.074976] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1327.079403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1329.120945] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1329.121329] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1329.123408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1331.802618] Lustre: server umount lustre-OST0000 complete [ 1334.240816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1334.241213] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1334.248172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1334.249827] Lustre: Skipped 2 previous similar messages [ 1339.202134] Lustre: server umount lustre-MDT0000 complete [ 1339.359960] LustreError: 82953:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1339.363778] LustreError: 82953:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 6 previous similar messages [ 1340.543184] Lustre: server umount lustre-MDT0001 complete [ 1341.796228] Lustre: server umount MGS complete [ 1345.272820] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 20:12:39 (1777594359) [ 1354.172717] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 1356.999331] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 1367.949956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1369.214500] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1373.090211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1373.180938] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1373.196117] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1373.232500] Lustre: lustre-MDT0001: new disk, initializing [ 1373.256439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1373.264543] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1373.266973] Lustre: 88499:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1373.269969] LustreError: 88499:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1374.508436] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1378.493343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1378.593649] Lustre: lustre-OST0000: new disk, initializing [ 1378.596103] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1378.613974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1380.388570] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1383.322428] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1384.927889] Lustre: 89908:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000002d54075c x1863942734382080/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1777594415 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1385.467814] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1385.472022] Lustre: MGS: Client 3455946e-4d9f-4217-a54c-62abcb33a556 (at 0@lo) reconnecting [ 1385.485352] Lustre: 90178:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1385.493688] LustreError: 88484:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1385.497435] LustreError: 88484:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1385.501857] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1385.537482] Lustre: lustre-MDT0000: new disk, initializing [ 1385.583506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1386.656309] Lustre: 90193:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000007fa809d6 x1863942734387584/t0(0) o700->lustre-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1777594416 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1386.661902] Lustre: 90193:0:(client.c:1637:after_reply()) Skipped 1 previous similar message [ 1386.795102] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1386.976594] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1389.986166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1389.988903] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1391.714068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 1391.715802] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1391.718018] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 1410.527923] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1410.530348] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1410.534026] Lustre: Skipped 1 previous similar message [ 1410.535777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1414.625803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1414.627488] Lustre: Skipped 1 previous similar message [ 1416.599651] Lustre: server umount lustre-OST0000 complete [ 1419.745835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1419.746357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1419.749032] LustreError: Skipped 1 previous similar message [ 1419.752035] Lustre: Skipped 1 previous similar message [ 1424.009815] Lustre: server umount lustre-MDT0000 complete [ 1424.863965] LustreError: 88468:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1424.870478] LustreError: 88468:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1431.401135] Lustre: server umount lustre-MDT0001 complete [ 1432.546674] Lustre: server umount MGS complete [ 1435.927097] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 20:14:10 (1777594450) [ 1441.415973] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 1442.447507] Key type lgssc unregistered [ 1442.568368] LNet: 92664:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1442.570565] LNetError: 92664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1442.579303] LNet: Removed LNI 192.168.204.125@tcp [ 1442.860487] Key type .llcrypt unregistered [ 1442.861559] Key type ._llcrypt unregistered [ 1449.375262] Key type ._llcrypt registered [ 1449.376259] Key type .llcrypt registered [ 1449.410334] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 1449.741743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1449.766558] alg: No test for adler32 (adler32-zlib) [ 1450.615498] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 1450.686219] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 1452.263112] Key type lgssc registered [ 1452.609241] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1455.112602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1455.120445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1456.281990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1457.662552] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1460.604310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1460.751318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1462.194247] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1464.006384] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1465.218632] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1467.496231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1467.608369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1469.500094] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1471.281948] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1480.937910] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 1486.306477] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1486.307133] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1486.307907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1486.309640] Lustre: Skipped 1 previous similar message [ 1486.313031] LustreError: Skipped 1 previous similar message [ 1490.127764] Lustre: server umount lustre-OST0000 complete [ 1491.425306] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1491.425531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1491.426061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1491.426069] Lustre: Skipped 1 previous similar message [ 1491.435210] Lustre: Skipped 1 previous similar message [ 1496.544204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1496.547362] Lustre: Skipped 2 previous similar messages [ 1497.490701] Lustre: server umount lustre-MDT0000 complete [ 1498.747212] LustreError: 93827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777594512 with bad export cookie 1241588305576762226 [ 1498.750102] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1498.752575] LustreError: 93827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1498.868862] Lustre: server umount lustre-MDT0001 complete [ 1502.275684] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 1503.291546] Key type lgssc unregistered [ 1503.419385] LNet: 97039:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1503.422141] LNetError: 97039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1503.428311] LNet: Removed LNI 192.168.204.125@tcp [ 1503.736061] Key type .llcrypt unregistered [ 1503.737302] Key type ._llcrypt unregistered [ 1509.917232] Key type ._llcrypt registered [ 1509.918402] Key type .llcrypt registered [ 1509.956500] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 1510.246404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1510.253439] alg: No test for adler32 (adler32-zlib) [ 1511.111369] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 1511.181244] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 1512.759165] Key type lgssc registered [ 1513.126275] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1515.039909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1515.039973] LustreError: 98083:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 1515.042259] Lustre: Skipped 1 previous similar message [ 1515.042268] LustreError: 98082:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 1515.049779] LustreError: 98083:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 1515.053190] LustreError: 98083:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1515.056083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1515.057971] LustreError: 98083:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1516.210293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1518.547478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1518.626767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1519.746517] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1521.113540] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing lsmod [ 1521.975739] LustreError: 99256:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 1521.975825] LustreError: 99255:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 1521.981939] LustreError: 99256:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 1521.985825] LustreError: 99256:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1521.989074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1521.991544] LustreError: 99256:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1522.102950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1522.930366] LustreError: 98150:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1522.934659] LustreError: 98150:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1522.936935] LustreError: 98150:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 1523.679851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1523.683033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1523.688524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1527.264273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1527.264945] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1527.266291] Lustre: Skipped 1 previous similar message [ 1527.269495] Lustre: Skipped 1 previous similar message [ 1528.289625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1528.292251] Lustre: Skipped 2 previous similar messages [ 1529.111227] Lustre: server umount lustre-MDT0000 complete [ 1530.330300] LustreError: 98097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777594544 with bad export cookie 2489544930446196168 [ 1530.333132] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1530.333915] LustreError: 98097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1530.337409] LustreError: 98553:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000053a0fce8 x1863942956662016/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1530.430210] Lustre: server umount lustre-MDT0001 complete [ 1541.656514] Lustre: server umount lustre-OST0000 complete [ 1544.320981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1544.448426] LustreError: 100316:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1544.456017] LustreError: 100316:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1544.468148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1545.677979] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1548.310880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1549.656556] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1551.358476] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1552.492383] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1554.657489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1554.756804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1554.758759] Lustre: Skipped 1 previous similar message [ 1555.816401] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 1556.531394] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1558.266084] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1561.055711] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1561.059418] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1561.066101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1564.131339] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1564.134504] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1566.603374] Lustre: server umount lustre-OST0000 complete [ 1569.248513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1569.248745] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1569.249266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1569.249271] Lustre: Skipped 2 previous similar messages [ 1569.256972] Lustre: Skipped 1 previous similar message [ 1579.488179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1579.490284] Lustre: Skipped 4 previous similar messages [ 1582.047123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1582.104501] Lustre: server umount lustre-MDT0000 complete [ 1583.396261] LustreError: 102468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777594597 with bad export cookie 2489544930446196847 [ 1583.400065] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1583.400808] LustreError: 102468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1583.533458] Lustre: server umount lustre-MDT0001 complete [ 1587.160353] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 1588.281738] Key type lgssc unregistered [ 1588.419466] LNet: 103491:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1588.422280] LNetError: 103491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1588.433451] LNet: Removed LNI 192.168.204.125@tcp [ 1588.748601] Key type .llcrypt unregistered [ 1588.749830] Key type ._llcrypt unregistered [ 1597.829609] Key type ._llcrypt registered [ 1597.830635] Key type .llcrypt registered [ 1597.862865] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 1598.166504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1598.173339] alg: No test for adler32 (adler32-zlib) [ 1599.020993] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 1599.096765] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 1600.671143] Key type lgssc registered [ 1601.027045] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1603.528451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1603.534315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1604.681509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1605.855457] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1608.471289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1608.586080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1609.820885] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1611.482679] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1612.629040] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1614.708816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1614.830197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1616.657923] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1617.895713] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 1618.553445] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1621.442214] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1622.204631] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1628.129637] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1628.131373] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1628.132726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1628.138234] Lustre: Skipped 1 previous similar message [ 1629.268181] Lustre: server umount lustre-OST0000 complete [ 1633.249618] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1633.250337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1633.253283] Lustre: Skipped 1 previous similar message [ 1633.259115] Lustre: Skipped 2 previous similar messages [ 1636.696831] Lustre: server umount lustre-MDT0000 complete [ 1637.908194] LustreError: 106482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777594652 with bad export cookie 16371761217417836903 [ 1637.911082] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1637.911716] LustreError: 106482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1638.021423] Lustre: server umount lustre-MDT0001 complete [ 1641.730801] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 1642.704898] Key type lgssc unregistered [ 1642.839368] LNet: 108147:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1642.841860] LNetError: 108147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1642.851277] LNet: Removed LNI 192.168.204.125@tcp [ 1643.140686] Key type .llcrypt unregistered [ 1643.141815] Key type ._llcrypt unregistered [ 1650.230236] Key type ._llcrypt registered [ 1650.231274] Key type .llcrypt registered [ 1650.268085] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 1650.615771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1650.675725] alg: No test for adler32 (adler32-zlib) [ 1651.532344] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 1651.608245] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 1653.183129] Key type lgssc registered [ 1653.547500] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1656.153648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1656.160901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1657.308176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1658.552860] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1661.388629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1661.515816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1662.799922] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1664.541444] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1665.728937] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1667.983939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1668.103767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1669.903099] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1671.712750] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1682.470101] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 1691.616159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1691.618655] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1691.623686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1692.641248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1692.641719] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1692.642883] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1696.207034] Lustre: server umount lustre-OST0000 complete [ 1697.759942] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1697.763769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1697.765685] Lustre: Skipped 1 previous similar message [ 1702.367860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1702.370612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1702.374550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1703.570605] Lustre: server umount lustre-MDT0000 complete [ 1704.792932] LustreError: 109311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777594718 with bad export cookie 6538379686509676931 [ 1704.796575] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1704.797910] LustreError: 109311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1704.920734] Lustre: server umount lustre-MDT0001 complete [ 1708.731864] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 1709.706675] Key type lgssc unregistered [ 1709.827386] LNet: 112522:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1709.830943] LNetError: 112522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1709.839423] LNet: Removed LNI 192.168.204.125@tcp [ 1710.160930] Key type .llcrypt unregistered [ 1710.162580] Key type ._llcrypt unregistered [ 1719.595265] Key type ._llcrypt registered [ 1719.597047] Key type .llcrypt registered [ 1719.637737] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 1719.959270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1719.972051] alg: No test for adler32 (adler32-zlib) [ 1720.835131] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 1720.922026] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 1722.511158] Key type lgssc registered [ 1722.898616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1725.554132] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1725.562337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1726.731687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1728.049179] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1730.978322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1731.103854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1732.465561] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1734.192653] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1735.379639] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1737.630683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1737.745989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1739.756089] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1741.538768] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1752.099959] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 1755.466675] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1756.763336] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1756.765477] Lustre: Skipped 2 previous similar messages [ 1758.195720] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1758.200217] Lustre: Skipped 2 previous similar messages [ 1762.809912] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1762.822383] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1766.714136] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.25@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1766.717484] Lustre: Skipped 2 previous similar messages [ 1769.408118] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.25@tcp 192.168.0.[2,10]@tcp [ 1771.814773] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.25/24@tcp in log lustre-MDT0000 [ 1771.817434] Lustre: Skipped 2 previous similar messages [ 1778.689038] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.25/24@tcp [ 1778.691472] Lustre: Skipped 1 previous similar message [ 1782.753675] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1782.754642] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1782.755293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1782.757098] LustreError: Skipped 1 previous similar message [ 1782.763703] Lustre: Skipped 1 previous similar message [ 1783.177777] Lustre: server umount lustre-OST0000 complete [ 1786.515730] Lustre: server umount lustre-MDT0000 complete [ 1787.360783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1787.363599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1787.368253] LustreError: 113899:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1787.373575] LustreError: 113899:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1787.765263] LustreError: 114615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777594801 with bad export cookie 7430104887674268030 [ 1787.769471] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1787.770790] LustreError: 114615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1787.896472] Lustre: server umount lustre-MDT0001 complete [ 1791.830281] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 1792.849703] Key type lgssc unregistered [ 1792.988390] LNet: 118685:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1792.991155] LNetError: 118685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1793.004302] LNet: Removed LNI 192.168.204.125@tcp [ 1793.310997] Key type .llcrypt unregistered [ 1793.312804] Key type ._llcrypt unregistered [ 1803.998849] Key type ._llcrypt registered [ 1803.999951] Key type .llcrypt registered [ 1804.048159] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 1804.542826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1804.570846] alg: No test for adler32 (adler32-zlib) [ 1805.433763] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 1805.521278] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 1807.111136] Key type lgssc registered [ 1807.456555] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1810.025375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1810.030791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1811.176960] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1811.182480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1811.187000] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.25/24@tcp [ 1812.466575] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1815.235975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1815.338842] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1815.347233] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.25/24@tcp [ 1815.362837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1816.666185] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1818.465903] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1819.653752] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1821.802017] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1821.915239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1823.802388] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1825.598426] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1836.263981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 1840.607737] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1840.610221] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1840.614852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1841.633182] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1841.633618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1841.633777] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1841.641822] Lustre: Skipped 1 previous similar message [ 1845.461897] Lustre: server umount lustre-OST0000 complete [ 1846.751817] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1846.756145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1851.360091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1851.362465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1851.366681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1856.991933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1856.993841] Lustre: Skipped 2 previous similar messages [ 1861.087101] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1861.145536] Lustre: server umount lustre-MDT0000 complete [ 1862.111901] LustreError: 120214:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1862.116648] LustreError: 120214:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 1862.394154] LustreError: 120193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777594876 with bad export cookie 16873693978598262412 [ 1862.397201] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1862.400065] LustreError: 120193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1862.515960] Lustre: server umount lustre-MDT0001 complete [ 1866.264342] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 1867.363745] Key type lgssc unregistered [ 1867.503463] LNet: 123467:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1867.506091] LNetError: 123467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1867.513502] LNet: Removed LNI 192.168.204.125@tcp [ 1867.874062] Key type .llcrypt unregistered [ 1867.875382] Key type ._llcrypt unregistered [ 1876.384423] Key type ._llcrypt registered [ 1876.385737] Key type .llcrypt registered [ 1876.419870] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 1882.746198] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 1883.238976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1883.255226] alg: No test for adler32 (adler32-zlib) [ 1884.112056] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 1884.212166] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 1885.799175] Key type lgssc registered [ 1886.168320] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1897.495359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1897.505932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1898.646977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1898.666393] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1898.723737] Lustre: lustre-MDT0000: new disk, initializing [ 1898.782247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1898.794535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1900.119084] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1904.257680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1904.300636] Lustre: 126805:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1904.327339] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1904.329690] Lustre: Skipped 1 previous similar message [ 1904.368271] Lustre: lustre-MDT0001: new disk, initializing [ 1904.403190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1904.417495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1904.423411] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1905.646336] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1908.795702] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1910.178386] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1912.517934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1912.629541] Lustre: lustre-OST0000: new disk, initializing [ 1912.631296] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1912.661746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1914.493880] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1917.574435] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1918.933234] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1921.016255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1921.020436] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1921.050817] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1922.065382] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 1923.399519] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1923.475017] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1926.116023] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1926.116760] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1926.118712] LustreError: Skipped 1 previous similar message [ 1926.119650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1926.123777] Lustre: Skipped 1 previous similar message [ 1926.128566] Lustre: Skipped 1 previous similar message [ 1930.258422] Lustre: server umount lustre-OST0000 complete [ 1935.327977] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1935.330150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1935.334989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1936.352335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1936.352904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1937.693774] Lustre: server umount lustre-MDT0000 complete [ 1939.021419] LustreError: 126798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777594953 with bad export cookie 5817014987314232069 [ 1939.024768] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1939.025747] LustreError: 126798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1939.146173] Lustre: server umount lustre-MDT0001 complete [ 1942.018688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1942.191220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1943.445198] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1946.276389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1946.384972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1947.753551] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1949.486326] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1950.716660] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1952.926659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1954.964871] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1956.762253] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1958.149940] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1968.410000] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 1969.666546] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1969.744326] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1973.142334] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1973.217076] Lustre: lustre-OST0001: new disk, initializing [ 1973.218900] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1973.255117] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1973.259246] Lustre: Skipped 1 previous similar message [ 1975.123346] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1978.297357] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1978.872951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1978.876397] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1978.902917] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1979.401062] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 1980.754666] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 1980.835279] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1982.044364] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a1b9878d000.ost_server_uuid 50 [ 1982.590282] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8a1b9878d000.ost_server_uuid in FULL state after 0 sec [ 1989.089949] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1989.091471] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1989.094947] LustreError: Skipped 1 previous similar message [ 1989.100704] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 1989.104906] Lustre: Skipped 2 previous similar messages [ 1990.592854] Lustre: server umount lustre-OST0001 complete [ 1992.159893] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1993.183899] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1993.189520] Lustre: Skipped 2 previous similar messages [ 1993.191367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1993.193938] Lustre: Skipped 1 previous similar message [ 1994.211381] LustreError: 132240:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1994.216188] LustreError: 132240:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 1997.968554] Lustre: server umount lustre-OST0000 complete [ 2004.448703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2004.451613] Lustre: Skipped 5 previous similar messages [ 2005.344319] Lustre: server umount lustre-MDT0000 complete [ 2006.530310] LustreError: 130626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777595020 with bad export cookie 5817014987314232937 [ 2006.532875] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2006.536181] LustreError: 130626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2006.661278] Lustre: server umount lustre-MDT0001 complete [ 2013.376960] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 2014.637859] Key type lgssc unregistered [ 2014.772690] LNet: 136554:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2014.776670] LNetError: 136554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2014.788513] LNet: Removed LNI 192.168.204.125@tcp [ 2015.130635] Key type .llcrypt unregistered [ 2015.131896] Key type ._llcrypt unregistered [ 2028.322061] Key type ._llcrypt registered [ 2028.323159] Key type .llcrypt registered [ 2028.359289] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 2034.364102] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 2034.648783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2034.664716] alg: No test for adler32 (adler32-zlib) [ 2035.514731] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 2035.591158] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 2037.167137] Key type lgssc registered [ 2037.515738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2048.748345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2048.760326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2049.878847] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2049.893524] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2049.935354] Lustre: lustre-MDT0000: new disk, initializing [ 2049.960896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2049.969222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2051.322211] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2056.092515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2056.134691] Lustre: 140649:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2056.168615] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2056.168687] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2056.174128] Lustre: Skipped 1 previous similar message [ 2056.212568] Lustre: lustre-MDT0001: new disk, initializing [ 2056.248090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2056.262178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2056.267163] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2057.655977] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2061.549552] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2062.919186] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2065.258648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2065.355964] Lustre: lustre-OST0000: new disk, initializing [ 2065.357615] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2065.384857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2067.207484] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2070.702589] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2075.126868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2075.131907] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2075.161155] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2077.846855] Lustre: Failing over lustre-OST0000 [ 2077.900451] Lustre: server umount lustre-OST0000 complete [ 2090.671321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2090.792627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2090.801137] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2092.429847] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2092.483975] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2093.673735] Lustre: Failing over lustre-MDT0000 [ 2093.768700] Lustre: server umount lustre-MDT0000 complete [ 2096.098844] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2096.106205] Lustre: Skipped 3 previous similar messages [ 2097.119209] Lustre: 138878:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777595094/real 1777595094] req@00000000bb35886f x1863943506128384/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1777595110 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2097.126626] Lustre: 138878:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2101.215203] Lustre: 138877:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777595099/real 1777595099] req@00000000e6c0adf0 x1863943506129536/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1777595115 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2101.216474] LustreError: 140661:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2101.228132] LustreError: 140661:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 7 previous similar messages [ 2102.662916] LustreError: 140658:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2106.336288] Lustre: 138879:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777595104/real 1777595104] req@0000000019e830eb x1863943506130688/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1777595120 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2106.338590] LustreError: 142398:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2106.351198] LustreError: 142398:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2107.096173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2107.149855] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2107.156064] Lustre: 140653:0:(mgc_request.c:1897:mgc_process_log()) MGC192.168.204.125@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2107.259708] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2107.283800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2107.315505] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2107.784295] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2108.865298] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2112.487479] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2112.497823] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2112.521390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2117.602425] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2117.603910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2117.604868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2117.608162] Lustre: Skipped 1 previous similar message [ 2117.613107] Lustre: Skipped 1 previous similar message [ 2119.627290] Lustre: server umount lustre-OST0000 complete [ 2122.720812] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2122.721124] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2122.721544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2122.725775] Lustre: Skipped 1 previous similar message [ 2122.728044] LustreError: Skipped 1 previous similar message [ 2126.927494] Lustre: server umount lustre-MDT0000 complete [ 2127.840179] LustreError: 141448:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2127.848194] LustreError: 141448:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2128.121969] LustreError: 140643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777595142 with bad export cookie 9751639255442522743 [ 2128.124483] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2128.125422] LustreError: 140643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2128.237135] Lustre: server umount lustre-MDT0001 complete [ 2131.567259] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 2132.548915] Key type lgssc unregistered [ 2132.683481] LNet: 145875:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2132.687441] LNetError: 145875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2132.701456] LNet: Removed LNI 192.168.204.125@tcp [ 2133.033332] Key type .llcrypt unregistered [ 2133.034471] Key type ._llcrypt unregistered [ 2152.744962] Key type ._llcrypt registered [ 2152.746115] Key type .llcrypt registered [ 2152.780434] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 2153.202735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2153.211386] alg: No test for adler32 (adler32-zlib) [ 2154.064864] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 2154.143030] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 2155.719218] Key type lgssc registered [ 2156.233112] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2159.929483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2159.941237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2161.236885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2163.288428] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2167.598594] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2170.244408] LustreError: 147404:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2170.255989] LustreError: 147404:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2170.260451] LustreError: 147404:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 2170.518435] Lustre: server umount lustre-MDT0000 complete [ 2178.584672] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 2184.014153] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 2202.233771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2202.356324] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2202.375902] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2202.428187] Lustre: lustre-MDT0000: new disk, initializing [ 2202.480730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2202.498403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2204.362977] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2209.830628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2209.880764] Lustre: 150556:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2209.921125] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2209.924060] Lustre: Skipped 1 previous similar message [ 2209.971243] Lustre: lustre-MDT0001: new disk, initializing [ 2210.020060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2210.041169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2210.048337] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2211.932497] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2215.917216] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2217.529839] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2220.536552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2220.654203] Lustre: lustre-OST0000: new disk, initializing [ 2220.658420] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2220.703364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2223.067770] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2226.363126] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2226.681518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2226.686306] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2226.718650] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2233.441449] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2234.048537] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 20:27:28 (1777595248) [ 2333.151610] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2333.158615] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2333.166271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2334.177676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2334.178095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2334.187448] Lustre: Skipped 1 previous similar message [ 2334.196626] Lustre: Skipped 1 previous similar message [ 2336.958731] Lustre: server umount lustre-MDT0000 complete [ 2338.653523] LustreError: 150546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777595352 with bad export cookie 13337256278776399975 [ 2338.655302] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2338.661977] LustreError: 150546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2339.296508] LustreError: 150562:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2339.298036] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2339.307021] LustreError: 150562:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2339.317376] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2343.329147] LustreError: 152851:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2343.329922] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2344.874194] Lustre: server umount lustre-MDT0001 complete [ 2352.464096] Lustre: server umount lustre-OST0000 complete [ 2356.353311] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 2359.965657] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 2373.158071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2373.240950] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2373.253972] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2373.295858] Lustre: lustre-MDT0000: new disk, initializing [ 2373.321963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2373.330806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2374.788768] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2378.938263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2378.976117] Lustre: 156782:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2378.998016] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2379.001025] Lustre: Skipped 1 previous similar message [ 2379.037834] Lustre: lustre-MDT0001: new disk, initializing [ 2379.078131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2379.083523] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2380.334868] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2383.367324] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2384.542537] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2386.737870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2386.853373] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2386.855892] Lustre: Skipped 1 previous similar message [ 2388.602432] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2390.712507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2390.716419] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2390.747978] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2391.549871] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2392.902316] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2392.977465] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2394.268050] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2394.337030] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2396.131762] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2396.132505] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2396.134822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2396.137774] Lustre: Skipped 1 previous similar message [ 2401.039086] Lustre: server umount lustre-OST0000 complete [ 2404.831750] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2404.834520] LustreError: Skipped 1 previous similar message [ 2404.836048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2404.841116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2404.843351] Lustre: Skipped 1 previous similar message [ 2408.340349] Lustre: server umount lustre-MDT0000 complete [ 2409.542074] LustreError: 156773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777595423 with bad export cookie 13337256278776558791 [ 2409.545110] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2409.546556] LustreError: 156773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2409.650038] Lustre: server umount lustre-MDT0001 complete [ 2412.173510] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 20:30:26 (1777595426) [ 2416.931328] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 2419.560099] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 2430.423805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2430.501977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2430.514510] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2430.517660] Lustre: Skipped 1 previous similar message [ 2430.555995] Lustre: lustre-MDT0000: new disk, initializing [ 2430.557505] Lustre: Skipped 1 previous similar message [ 2430.580702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2430.586746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2431.889289] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2435.951228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2435.988118] Lustre: 163085:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2436.087812] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2437.280362] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2440.220551] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2441.325577] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2443.507706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2444.467958] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2444.498147] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2445.496437] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2448.516749] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2455.010096] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2455.010430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2455.011310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2455.011313] Lustre: Skipped 2 previous similar messages [ 2455.012271] LustreError: Skipped 1 previous similar message [ 2455.015432] Lustre: Skipped 2 previous similar messages [ 2456.459990] Lustre: server umount lustre-OST0000 complete [ 2463.765906] Lustre: server umount lustre-MDT0000 complete [ 2465.018992] LustreError: 163868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777595479 with bad export cookie 13337256278776559659 [ 2465.021829] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2465.023309] LustreError: 163868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2465.127374] Lustre: server umount lustre-MDT0001 complete [ 2467.812988] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 20:31:21 (1777595481) [ 2472.832525] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 2475.580206] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 2486.947417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2487.039315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2487.049469] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2487.051439] Lustre: Skipped 3 previous similar messages [ 2487.086976] Lustre: lustre-MDT0000: new disk, initializing [ 2487.089010] Lustre: Skipped 2 previous similar messages [ 2487.111858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2487.115039] Lustre: Skipped 2 previous similar messages [ 2487.120966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2487.123128] Lustre: Skipped 2 previous similar messages [ 2488.345294] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2492.219162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2492.257303] Lustre: 169002:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2492.346647] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2493.485881] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2496.419077] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2497.547594] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2499.666337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2501.511643] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2504.366347] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2505.746455] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2515.938851] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2515.939064] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2515.941452] LustreError: Skipped 1 previous similar message [ 2515.945516] Lustre: Skipped 2 previous similar messages [ 2515.947992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2515.951236] Lustre: Skipped 4 previous similar messages [ 2517.518421] Lustre: server umount lustre-OST0000 complete [ 2524.815752] Lustre: server umount lustre-MDT0000 complete [ 2525.946242] LustreError: 168993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777595540 with bad export cookie 13337256278776560730 [ 2525.949241] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2525.950737] LustreError: 168993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2526.064109] Lustre: server umount lustre-MDT0001 complete [ 2529.334714] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 2530.293642] Key type lgssc unregistered [ 2530.422363] LNet: 172474:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2530.424873] LNetError: 172474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2530.436361] LNet: Removed LNI 192.168.204.125@tcp [ 2530.720106] Key type .llcrypt unregistered [ 2530.721220] Key type ._llcrypt unregistered [ 2539.074171] Key type ._llcrypt registered [ 2539.075827] Key type .llcrypt registered [ 2539.109608] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 2539.487181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2539.493408] alg: No test for adler32 (adler32-zlib) [ 2540.329630] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 2540.404704] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 2541.975119] Key type lgssc registered [ 2542.327435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2544.844058] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2544.852109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2546.023713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2547.290712] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2550.070432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2550.215994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2551.624917] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2553.479378] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2554.736594] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2556.959317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2557.074791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2559.015182] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2560.808450] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2571.429336] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2591.201062] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2591.204171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2591.209701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2592.225960] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2592.226262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2592.226914] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2592.234529] Lustre: Skipped 1 previous similar message [ 2593.934177] Lustre: server umount lustre-OST0000 complete [ 2596.319877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2596.322691] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2596.327250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2601.310981] Lustre: server umount lustre-MDT0000 complete [ 2602.464932] LustreError: 173850:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2602.469906] LustreError: 173850:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 2602.630405] LustreError: 173829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777595616 with bad export cookie 4785139535154627244 [ 2602.632962] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2602.634449] LustreError: 173829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2602.759863] Lustre: server umount lustre-MDT0001 complete [ 2606.386179] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 2607.509442] Key type lgssc unregistered [ 2607.644382] LNet: 177046:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2607.646739] LNetError: 177046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2607.655320] LNet: Removed LNI 192.168.204.125@tcp [ 2608.000549] Key type .llcrypt unregistered [ 2608.001663] Key type ._llcrypt unregistered [ 2617.083874] Key type ._llcrypt registered [ 2617.085009] Key type .llcrypt registered [ 2617.128982] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 2617.564788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2617.591671] alg: No test for adler32 (adler32-zlib) [ 2618.438622] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 2618.519188] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 2620.095109] Key type lgssc registered [ 2620.452270] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2622.978183] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2622.985172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2624.138700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2625.446467] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2628.273682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2628.392215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2629.739904] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2631.551256] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2632.806426] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2635.050490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2635.160611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2637.032295] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2638.825402] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2649.512649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2658.783813] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2658.786705] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2658.790878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2659.727253] Lustre: lustre-OST0000: Not available for connect from 192.168.204.25@tcp (stopping) [ 2659.809246] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2659.809580] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2662.862820] Lustre: server umount lustre-OST0000 complete [ 2664.584670] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8a1b88976000.ost_server_uuid 50 [ 2665.087909] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8a1b88976000.ost_server_uuid in DISCONN state after 0 sec [ 2665.617474] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2679.775929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2679.778350] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2679.782484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2679.784243] Lustre: Skipped 2 previous similar messages [ 2681.753907] Lustre: server umount lustre-MDT0000 complete [ 2682.981081] LustreError: 179138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777595697 with bad export cookie 8282619476029271787 [ 2682.984623] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2682.984907] LustreError: 179138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2683.101414] Lustre: server umount lustre-MDT0001 complete [ 2685.795105] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 20:34:59 (1777595699) [ 2687.996158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2688.150275] LustreError: 181946:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2688.158862] LustreError: 181946:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2688.174210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2689.512337] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2692.347727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2692.488915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2693.901899] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2695.773188] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2697.047129] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2699.318793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2701.296021] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2703.064328] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2705.400362] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2705.474765] Lustre: lustre-OST0001: new disk, initializing [ 2705.476425] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2705.509098] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2705.512365] Lustre: Skipped 1 previous similar message [ 2707.361693] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2707.839997] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2707.844347] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2707.879892] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2710.536596] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2713.061897] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2713.065121] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2713.066111] Lustre: Skipped 1 previous similar message [ 2713.071598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2713.073895] Lustre: Skipped 2 previous similar messages [ 2717.560792] Lustre: server umount lustre-OST0000 complete [ 2719.202744] LustreError: 183538:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2719.208508] LustreError: 183538:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2719.639629] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2719.726307] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2721.060426] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2721.138542] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2721.160676] LustreError: 183539:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2724.321932] LustreError: 183540:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2724.327702] LustreError: 183540:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2729.440993] LustreError: 183540:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2729.445878] LustreError: 183540:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2733.537097] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2733.539930] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2733.545665] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2733.547838] Lustre: Skipped 4 previous similar messages [ 2737.234675] Lustre: server umount lustre-OST0001 complete [ 2744.666796] Lustre: server umount lustre-MDT0000 complete [ 2744.799970] LustreError: 181965:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2744.804745] LustreError: 181965:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 2745.872596] LustreError: 181926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777595759 with bad export cookie 8282619476029272844 [ 2745.876755] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2745.879245] LustreError: 181926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2746.004724] Lustre: server umount lustre-MDT0001 complete [ 2753.949645] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 20:36:08 (1777595768) [ 2756.172391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2756.285879] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2756.298318] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2756.380326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2757.698656] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2760.515675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2760.548956] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2760.562145] Lustre: 187709:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2761.935089] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2763.749079] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2764.995732] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2767.276087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2767.346424] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2769.323475] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2769.871730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2771.120927] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2773.313260] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2773.337732] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2775.402183] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2776.537737] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 2777.310383] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2780.128412] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2780.128588] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2780.131898] LustreError: Skipped 3 previous similar messages [ 2780.135938] Lustre: Skipped 3 previous similar messages [ 2780.136636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2780.141366] Lustre: Skipped 6 previous similar messages [ 2784.254791] Lustre: server umount lustre-OST0000 complete [ 2786.785741] LustreError: 189772:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2786.791250] LustreError: 189772:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2797.027542] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2797.030701] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2797.032768] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2797.033711] Lustre: Skipped 2 previous similar messages [ 2797.035313] Lustre: Skipped 3 previous similar messages [ 2801.037640] Lustre: server umount lustre-OST0001 complete [ 2808.409713] Lustre: server umount lustre-MDT0000 complete [ 2809.622211] LustreError: 188441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777595823 with bad export cookie 8282619476029274006 [ 2809.624605] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2809.625630] LustreError: 188441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2809.749870] Lustre: server umount lustre-MDT0001 complete [ 2817.550629] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 20:37:11 (1777595831) [ 2822.724597] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 2825.861619] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 2837.736744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2837.835596] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2837.849290] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2837.898838] Lustre: lustre-MDT0000: new disk, initializing [ 2837.926500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2837.928741] Lustre: Skipped 3 previous similar messages [ 2837.934384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2839.296553] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2843.527903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2843.567433] Lustre: 195286:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2843.591530] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2843.593622] Lustre: Skipped 1 previous similar message [ 2843.642351] Lustre: lustre-MDT0001: new disk, initializing [ 2843.686813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2843.690615] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2845.074872] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2848.263996] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2849.517955] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2851.913052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2852.017204] Lustre: lustre-OST0000: new disk, initializing [ 2852.019063] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2853.953522] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2857.077311] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2858.426835] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2860.022796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2860.026176] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2860.050066] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2860.532036] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2861.886739] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2861.970790] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2865.122435] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2865.122901] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2865.127379] Lustre: Skipped 2 previous similar messages [ 2865.128156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2865.130674] LustreError: Skipped 3 previous similar messages [ 2865.132567] Lustre: Skipped 4 previous similar messages [ 2868.758950] Lustre: server umount lustre-OST0000 complete [ 2876.191634] Lustre: server umount lustre-MDT0000 complete [ 2877.456421] LustreError: 196065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777595891 with bad export cookie 8282619476029275770 [ 2877.458565] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2877.462028] LustreError: 196065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2877.575930] Lustre: server umount lustre-MDT0001 complete [ 2880.517018] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2880.699857] LustreError: 199111:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2880.706133] LustreError: 199111:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 6 previous similar messages [ 2882.094950] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2884.947801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2886.401853] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2888.159826] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2889.383446] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2891.621517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2893.635815] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2895.415187] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2896.793159] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2906.049524] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 2907.461509] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2907.546522] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2914.331666] Lustre: server umount lustre-OST0000 complete [ 2916.277357] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2917.683210] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2941.313761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2941.436781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2941.439703] Lustre: Skipped 5 previous similar messages [ 2943.253072] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2945.035510] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2945.596975] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2950.753822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2950.756577] Lustre: Skipped 7 previous similar messages [ 2952.719312] Lustre: server umount lustre-OST0000 complete [ 2956.256645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2956.262832] Lustre: Skipped 6 previous similar messages [ 2960.083964] Lustre: server umount lustre-MDT0000 complete [ 2961.344675] LustreError: 199092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777595975 with bad export cookie 8282619476029276624 [ 2961.348515] LustreError: 199092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2961.349450] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2961.478242] Lustre: server umount lustre-MDT0001 complete [ 2964.303556] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 20:39:38 (1777595978) [ 2966.502838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2966.661830] LustreError: 204610:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2967.986968] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2970.755189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2972.291980] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2974.029576] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2975.270551] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2977.499041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2979.506319] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2981.251312] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2982.567815] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2991.823448] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 2993.151681] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2993.231259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2995.417809] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2995.485825] Lustre: lustre-OST0001: new disk, initializing [ 2995.488031] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2997.348717] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3000.411432] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3001.817714] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3003.909700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3003.913043] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3003.938124] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3004.952243] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3006.200668] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3006.268815] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3008.999978] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3009.004218] LustreError: Skipped 2 previous similar messages [ 3012.988625] Lustre: server umount lustre-OST0001 complete [ 3014.920654] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3014.993167] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3016.279435] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3016.353700] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3039.944644] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3041.821275] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3043.548214] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3044.114149] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3050.873406] Lustre: server umount lustre-OST0001 complete [ 3077.087177] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3077.142558] Lustre: server umount lustre-OST0000 complete [ 3081.184545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3081.186757] Lustre: Skipped 19 previous similar messages [ 3084.635644] Lustre: server umount lustre-MDT0000 complete [ 3085.883075] LustreError: 204636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777596099 with bad export cookie 8282619476029277555 [ 3085.884965] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3085.886982] LustreError: 204636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3092.097442] Lustre: server umount lustre-MDT0001 complete [ 3099.532147] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 20:41:53 (1777596113) [ 3101.635074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3101.741437] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3101.751404] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3101.834316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3101.836357] Lustre: Skipped 5 previous similar messages [ 3103.135223] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3105.919543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3105.952273] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3105.961949] Lustre: 212629:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3107.344853] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3109.079569] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3110.253812] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3112.468327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3112.541189] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3114.503675] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3116.316868] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3120.087243] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3122.073014] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3123.796152] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3127.397948] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 3128.689805] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3128.760131] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3130.003367] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3130.075284] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3131.236884] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a1b85434000.ost_server_uuid 50 [ 3131.728457] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8a1b85434000.ost_server_uuid in FULL state after 0 sec [ 3132.067882] Lustre: Permanently deactivating lustre-OST0001 [ 3132.074177] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3132.886546] Lustre: Permanently reactivating lustre-OST0001 [ 3136.996322] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3136.999782] Lustre: Skipped 7 previous similar messages [ 3139.899201] Lustre: server umount lustre-OST0001 complete [ 3141.600055] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3141.602634] LustreError: Skipped 4 previous similar messages [ 3142.119683] LustreError: 214247:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3142.125841] LustreError: 214247:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 28 previous similar messages [ 3147.210069] Lustre: server umount lustre-OST0000 complete [ 3154.579081] Lustre: server umount lustre-MDT0000 complete [ 3155.826296] LustreError: 212620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777596169 with bad export cookie 8282619476029278815 [ 3155.830167] LustreError: 212620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3155.830240] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3155.949829] Lustre: server umount lustre-MDT0001 complete [ 3163.752902] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 20:42:57 (1777596177) [ 3166.225729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3166.334164] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3166.348839] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3166.354270] Lustre: Skipped 5 previous similar messages [ 3167.801863] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3170.705565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3170.738162] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3170.740220] Lustre: Skipped 1 previous similar message [ 3170.750072] Lustre: 218729:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3172.329967] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3174.182718] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3175.415784] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3177.586489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3177.695690] Lustre: Permanently deactivating lustre-OST0000 [ 3177.701724] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3177.830147] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3179.611684] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3181.348935] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3183.560193] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3185.438490] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3185.940293] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3187.240307] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3187.789522] Lustre: Permanently reactivating lustre-OST0000 [ 3195.380128] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3195.382715] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3195.386411] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3195.399961] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3195.405886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3202.756229] Lustre: server umount lustre-OST0001 complete [ 3218.399133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3218.454097] Lustre: server umount lustre-OST0000 complete [ 3234.271089] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3234.342467] Lustre: server umount lustre-MDT0000 complete [ 3235.588206] LustreError: 218722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777596249 with bad export cookie 8282619476029280446 [ 3235.591566] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3235.594772] LustreError: 218722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3235.715085] Lustre: server umount lustre-MDT0001 complete [ 3239.562284] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 3240.605862] Key type lgssc unregistered [ 3240.728349] LNet: 223114:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3240.731483] LNetError: 223114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3240.743315] LNet: Removed LNI 192.168.204.125@tcp [ 3241.050939] Key type .llcrypt unregistered [ 3241.052233] Key type ._llcrypt unregistered [ 3249.434479] Key type ._llcrypt registered [ 3249.435587] Key type .llcrypt registered [ 3249.479310] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 3249.988260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3250.001172] alg: No test for adler32 (adler32-zlib) [ 3250.860479] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 3250.961167] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 3252.535131] Key type lgssc registered [ 3252.905270] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3256.222958] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3256.228878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3257.405914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3258.760327] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3261.672647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3261.715839] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3261.727918] Lustre: 224530:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3261.734281] Lustre: Permanently deactivating lustre-MDT0001 [ 3261.741576] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3261.857866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3263.201401] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3264.109509] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3264.113078] LustreError: 224573:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3264.116672] LustreError: 224573:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3264.119513] LustreError: 224573:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 1, failed: rc = -108 [ 3265.006180] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3266.179180] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3268.403735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3268.537843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3270.466515] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3271.587865] LustreError: 226156:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3271.594446] LustreError: 226156:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 3272.250828] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3274.490347] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3274.569511] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3276.479653] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3278.549300] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3279.852627] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3281.896163] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3281.897168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3284.276688] Lustre: Permanently reactivating lustre-MDT0001 [ 3284.282025] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3284.284254] Lustre: Skipped 2 previous similar messages [ 3288.270508] Lustre: lustre-MDT0001: Client d07fef3c-2fd4-4b4a-9ff7-a56c2371e8cc (at 192.168.204.25@tcp) reconnecting [ 3289.579975] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3290.159104] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3292.297121] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3292.392747] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3293.808658] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3293.906141] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3294.755301] Lustre: Permanently deactivating lustre-MDT0001 [ 3294.761828] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3294.764118] Lustre: Skipped 2 previous similar messages [ 3296.647674] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3305.378103] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 3305.381852] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3305.388144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3309.536968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3309.538608] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3309.544536] Lustre: Skipped 2 previous similar messages [ 3311.578849] Lustre: server umount lustre-MDT0000 complete [ 3312.821366] LustreError: 224523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777596326 with bad export cookie 7472781747619125585 [ 3312.824049] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3312.825467] LustreError: 224523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3312.947118] Lustre: server umount lustre-MDT0001 complete [ 3320.315614] Lustre: server umount lustre-OST0000 complete [ 3323.869728] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 20:45:37 (1777596337) [ 3327.330641] Lustre: server umount lustre-OST0001 complete [ 3329.333465] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 3332.491569] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 3343.957144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3344.050408] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3344.053970] Lustre: Skipped 2 previous similar messages [ 3344.064248] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3344.111266] Lustre: lustre-MDT0000: new disk, initializing [ 3344.139796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3344.148938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3345.522632] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3349.689667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3349.735277] Lustre: 232169:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3349.765522] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3349.767551] Lustre: Skipped 1 previous similar message [ 3349.810043] Lustre: lustre-MDT0001: new disk, initializing [ 3349.856274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3349.859590] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3351.181159] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3354.255563] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3355.439441] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3357.640868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3357.761738] Lustre: lustre-OST0000: new disk, initializing [ 3357.763267] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3357.795603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3357.798263] Lustre: Skipped 1 previous similar message [ 3358.391913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3358.397197] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3358.435972] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3359.713944] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3362.742158] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3363.342792] LustreError: 233136:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3367.568479] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3367.634188] Lustre: lustre-OST0001: new disk, initializing [ 3367.636129] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3367.905304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3367.910493] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3367.945873] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3369.646088] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3373.028014] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3373.431143] LustreError: 233136:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3373.449243] LustreError: 234567:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3373.630673] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3383.543092] LustreError: 234567:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3385.742722] Lustre: lustre-OST0001: Not available for connect from 192.168.204.25@tcp (stopping) [ 3385.744525] Lustre: Skipped 3 previous similar messages [ 3388.386100] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3388.388926] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3388.390080] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3388.392908] Lustre: Skipped 1 previous similar message [ 3388.396314] Lustre: Skipped 1 previous similar message [ 3390.272600] Lustre: server umount lustre-OST0001 complete [ 3390.856778] LustreError: 233894:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3393.511029] LustreError: 233895:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3393.516052] LustreError: 233895:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3398.628722] LustreError: 234966:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3398.635112] LustreError: 234966:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3403.746895] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3403.750301] LustreError: 234966:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3403.750521] Lustre: Skipped 1 previous similar message [ 3403.750819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3403.755345] LustreError: 234966:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3408.397398] Lustre: server umount lustre-OST0000 complete [ 3411.424491] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3411.428894] LustreError: Skipped 1 previous similar message [ 3411.431758] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3413.985649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3413.990207] Lustre: Skipped 3 previous similar messages [ 3415.920258] Lustre: server umount lustre-MDT0000 complete [ 3417.406487] LustreError: 232946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777596431 with bad export cookie 7472781747619128182 [ 3417.411839] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3417.412663] LustreError: 232946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3423.613653] Lustre: server umount lustre-MDT0001 complete [ 3427.226586] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 3428.320598] Key type lgssc unregistered [ 3428.466466] LNet: 236814:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3428.471273] LNetError: 236814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3428.485466] LNet: Removed LNI 192.168.204.125@tcp [ 3428.822293] Key type .llcrypt unregistered [ 3428.824217] Key type ._llcrypt unregistered [ 3442.835892] Key type ._llcrypt registered [ 3442.836973] Key type .llcrypt registered [ 3442.876260] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 3443.349418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3443.372077] alg: No test for adler32 (adler32-zlib) [ 3444.220714] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 3444.305562] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 3445.887098] Key type lgssc registered [ 3446.261289] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3448.835646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3448.841943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3449.939730] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3449.948820] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3450.020819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3451.355199] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3454.105960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3454.137761] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3454.148630] Lustre: 238980:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3454.235068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3455.527135] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3457.258389] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3458.468178] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3460.750394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3460.823530] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3460.900086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3462.818868] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3463.261641] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 3464.605852] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3504.100497] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3504.102398] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3504.110497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3504.118478] Lustre: Skipped 1 previous similar message [ 3509.217842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3509.220024] Lustre: Skipped 1 previous similar message [ 3509.606481] Lustre: server umount lustre-OST0000 complete [ 3511.124448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3515.325926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3515.452062] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3517.342696] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3519.107754] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3534.824374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3534.828974] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3534.829200] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3555.298470] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3555.298625] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3555.299822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3555.301961] LustreError: Skipped 1 previous similar message [ 3555.309585] Lustre: Skipped 2 previous similar messages [ 3558.364356] Lustre: server umount lustre-OST0000 complete [ 3560.416627] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3560.417283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3560.420474] Lustre: Skipped 1 previous similar message [ 3560.422821] Lustre: Skipped 1 previous similar message [ 3565.536382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3565.538756] Lustre: Skipped 2 previous similar messages [ 3565.719169] Lustre: server umount lustre-MDT0000 complete [ 3566.997281] LustreError: 239703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777596581 with bad export cookie 3862567661359358568 [ 3567.000668] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3567.001447] LustreError: 239703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3567.123177] Lustre: server umount lustre-MDT0001 complete [ 3570.738716] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 3571.760542] Key type lgssc unregistered [ 3571.872376] LNet: 243924:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3571.874980] LNetError: 243924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3571.884346] LNet: Removed LNI 192.168.204.125@tcp [ 3572.166678] Key type .llcrypt unregistered [ 3572.167823] Key type ._llcrypt unregistered [ 3580.788129] Key type ._llcrypt registered [ 3580.789139] Key type .llcrypt registered [ 3580.828602] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 3581.205652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3581.255445] alg: No test for adler32 (adler32-zlib) [ 3582.118486] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 3582.206738] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 3583.791219] Key type lgssc registered [ 3584.184491] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3586.781681] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3586.789782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3587.965848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3589.253301] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3592.022979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3592.148183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3593.511227] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3595.242078] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3596.402758] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3598.689762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3598.810900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3600.726631] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3602.516190] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3613.159263] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3627.488434] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3627.492488] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3627.498481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3628.515106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3628.516611] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3628.517697] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3631.766608] Lustre: server umount lustre-OST0000 complete [ 3633.633113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3633.642275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3633.647478] Lustre: Skipped 1 previous similar message [ 3638.241129] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3638.246550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3638.256176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3639.522266] Lustre: server umount lustre-MDT0000 complete [ 3640.909412] LustreError: 246916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777596655 with bad export cookie 3822368106491211301 [ 3640.912986] LustreError: 246916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3640.913437] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3641.034502] Lustre: server umount lustre-MDT0001 complete [ 3644.707609] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 3645.786862] Key type lgssc unregistered [ 3645.909387] LNet: 249269:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3645.911691] LNetError: 249269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3645.921283] LNet: Removed LNI 192.168.204.125@tcp [ 3646.239096] Key type .llcrypt unregistered [ 3646.241244] Key type ._llcrypt unregistered [ 3653.766623] Key type ._llcrypt registered [ 3653.767646] Key type .llcrypt registered [ 3653.800213] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 3654.102094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3654.162595] alg: No test for adler32 (adler32-zlib) [ 3655.007990] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 3655.100883] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 3656.687127] Key type lgssc registered [ 3657.003958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3659.473955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3659.481742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3660.647935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3661.812422] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3664.533595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3664.658204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3665.910279] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3667.698648] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3668.889478] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3671.114263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3671.243458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3673.115935] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3674.909364] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3686.631456] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3695.071910] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3695.074678] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3695.079539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3697.121626] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3697.122616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3697.124304] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3697.130620] Lustre: Skipped 1 previous similar message [ 3700.125295] Lustre: server umount lustre-OST0000 complete [ 3702.241908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3702.243655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3702.250451] Lustre: Skipped 1 previous similar message [ 3702.257426] Lustre: Skipped 1 previous similar message [ 3707.360465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3707.363194] Lustre: Skipped 1 previous similar message [ 3707.741680] Lustre: server umount lustre-MDT0000 complete [ 3709.247219] LustreError: 250431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777596723 with bad export cookie 8642679666160245673 [ 3709.252973] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3709.255668] LustreError: 250431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3709.443938] Lustre: server umount lustre-MDT0001 complete [ 3713.403914] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 3714.555199] Key type lgssc unregistered [ 3714.704497] LNet: 253787:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3714.709076] LNetError: 253787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3714.718562] LNet: Removed LNI 192.168.204.125@tcp [ 3715.054202] Key type .llcrypt unregistered [ 3715.055713] Key type ._llcrypt unregistered [ 3723.851395] Key type ._llcrypt registered [ 3723.852527] Key type .llcrypt registered [ 3723.893194] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 3724.314838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3724.388790] alg: No test for adler32 (adler32-zlib) [ 3725.269566] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 3725.367742] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 3726.959158] Key type lgssc registered [ 3727.398080] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3730.099996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3730.107182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3731.296864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3732.613682] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3735.583481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3735.712363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3737.178712] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3739.129668] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3740.550601] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3743.348071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3743.523094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3745.652134] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3747.529776] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3757.868035] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3768.291847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3768.293799] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3768.300032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3768.307745] Lustre: Skipped 1 previous similar message [ 3772.508413] Lustre: server umount lustre-OST0000 complete [ 3776.992812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3776.997921] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3777.005325] Lustre: Skipped 1 previous similar message [ 3777.008403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3778.528881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3778.529541] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3780.197873] Lustre: server umount lustre-MDT0000 complete [ 3781.495670] LustreError: 255144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777596795 with bad export cookie 5376470487751927578 [ 3781.499332] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3781.502327] LustreError: 255144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3781.624468] Lustre: server umount lustre-MDT0001 complete [ 3785.325744] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 3786.495752] Key type lgssc unregistered [ 3786.642426] LNet: 259131:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3786.646555] LNetError: 259131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3786.657311] LNet: Removed LNI 192.168.204.125@tcp [ 3786.976924] Key type .llcrypt unregistered [ 3786.978557] Key type ._llcrypt unregistered [ 3793.759446] Key type ._llcrypt registered [ 3793.760631] Key type .llcrypt registered [ 3793.798120] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 3794.136075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3794.158078] alg: No test for adler32 (adler32-zlib) [ 3795.013835] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 3795.104279] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 3796.695174] Key type lgssc registered [ 3797.095785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3799.786955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3799.793972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3800.992855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3802.283875] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3805.118080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3805.251074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3806.576980] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3808.630348] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3810.099671] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3812.543803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3812.691143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3814.778816] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3816.568899] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3826.023715] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 3831.265183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3831.266770] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3831.268992] LustreError: Skipped 1 previous similar message [ 3831.273573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3831.277184] Lustre: Skipped 1 previous similar message [ 3832.728859] Lustre: server umount lustre-OST0000 complete [ 3835.872685] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3835.879099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3835.887791] Lustre: Skipped 1 previous similar message [ 3835.891427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3836.147576] Lustre: server umount lustre-MDT0000 complete [ 3836.386380] LustreError: 260338:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3836.397664] LustreError: 260338:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 3838.108438] LustreError: 262679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777596852 with bad export cookie 18125837377092991446 [ 3838.116421] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3838.117345] LustreError: 262679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3838.300272] Lustre: server umount lustre-MDT0001 complete [ 3842.297316] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 3843.308355] Key type lgssc unregistered [ 3843.432464] LNet: 263700:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3843.436154] LNetError: 263700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3843.445352] LNet: Removed LNI 192.168.204.125@tcp [ 3843.767423] Key type .llcrypt unregistered [ 3843.768704] Key type ._llcrypt unregistered [ 3847.367863] Key type ._llcrypt registered [ 3847.369065] Key type .llcrypt registered [ 3847.406616] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 3852.856653] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 3859.021477] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 3859.371814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3859.427136] alg: No test for adler32 (adler32-zlib) [ 3860.273637] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 3860.355587] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 3861.943160] Key type lgssc registered [ 3862.296347] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3873.625570] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3873.631023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3874.729327] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3874.740840] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3874.778584] Lustre: lustre-MDT0000: new disk, initializing [ 3874.803144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3874.810514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3876.141215] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3880.315736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3880.359212] Lustre: 267173:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3880.386613] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3880.388991] Lustre: Skipped 1 previous similar message [ 3880.427085] Lustre: lustre-MDT0001: new disk, initializing [ 3880.463044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3880.474802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3880.478085] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3881.796364] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3884.849996] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3886.030617] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3888.376617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3888.488910] Lustre: lustre-OST0000: new disk, initializing [ 3888.491481] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3888.522821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3890.433681] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3891.905245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3891.910821] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3891.949850] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3893.788411] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3895.280534] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3895.347467] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3896.825643] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3896.939424] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3902.437597] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3902.438306] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3902.439822] LustreError: Skipped 1 previous similar message [ 3902.445061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3902.448970] Lustre: Skipped 1 previous similar message [ 3903.757744] Lustre: server umount lustre-OST0000 complete [ 3906.015916] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3906.018739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3906.023160] Lustre: Skipped 1 previous similar message [ 3906.025187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3907.552120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3907.552789] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3912.671942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3912.674105] Lustre: Skipped 2 previous similar messages [ 3917.792023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3917.795670] Lustre: Skipped 1 previous similar message [ 3919.327104] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3919.402147] Lustre: server umount lustre-MDT0000 complete [ 3920.647187] LustreError: 267167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777596934 with bad export cookie 9600900809072656342 [ 3920.650164] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3920.650913] LustreError: 267167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3920.760013] Lustre: server umount lustre-MDT0001 complete [ 3923.515740] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 20:55:37 (1777596937) [ 3925.830947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3926.011800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3927.303402] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3930.124074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3930.246070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3931.481937] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3933.231196] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3934.387707] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3936.674777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3936.739714] LustreError: 272770:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3936.746173] LustreError: 272770:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 3938.690220] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3940.457246] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3951.077866] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3951.081944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3951.090289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3951.093493] Lustre: Skipped 1 previous similar message [ 3955.597275] Lustre: server umount lustre-OST0000 complete [ 3961.312797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3961.312857] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3961.315225] LustreError: Skipped 1 previous similar message [ 3961.318951] Lustre: Skipped 2 previous similar messages [ 3963.177460] Lustre: server umount lustre-MDT0000 complete [ 3964.459615] LustreError: 271159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777596978 with bad export cookie 9600900809072657210 [ 3964.464526] LustreError: 271159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3964.465336] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3964.590720] Lustre: server umount lustre-MDT0001 complete [ 3968.147660] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 3969.199338] Key type lgssc unregistered [ 3969.340496] LNet: 274355:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3969.343201] LNetError: 274355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3969.349484] LNet: Removed LNI 192.168.204.125@tcp [ 3969.668759] Key type .llcrypt unregistered [ 3969.669814] Key type ._llcrypt unregistered [ 3978.125864] Key type ._llcrypt registered [ 3978.127128] Key type .llcrypt registered [ 3978.164629] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 3984.311475] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 3984.773666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3984.780339] alg: No test for adler32 (adler32-zlib) [ 3985.633701] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 3985.715370] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 3987.295127] Key type lgssc registered [ 3987.667825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4003.405477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4003.413540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4004.534608] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4004.555450] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4004.610355] Lustre: lustre-MDT0000: new disk, initializing [ 4004.656626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4004.667919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4006.156852] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4010.435937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4010.484405] Lustre: 278118:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4010.509289] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4010.511118] Lustre: Skipped 1 previous similar message [ 4010.549119] Lustre: lustre-MDT0001: new disk, initializing [ 4010.582391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4010.595626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4010.599782] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4011.909125] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4014.998413] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4016.143795] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4018.533051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4018.657765] Lustre: lustre-OST2710: new disk, initializing [ 4018.660122] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4018.702490] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4019.336235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4019.342118] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4019.391353] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4021.086837] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4024.332361] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4026.632955] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4026.691631] Lustre: lustre-OST03e8: new disk, initializing [ 4026.693387] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4026.724777] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4028.603758] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4029.054566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4029.059248] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4029.085293] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4030.344077] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4032.958639] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4033.047611] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4034.564097] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4034.654334] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4036.064788] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4036.151521] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4037.592684] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4037.689576] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4039.654426] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4039.656807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4039.657930] Lustre: Skipped 3 previous similar messages [ 4039.660041] Lustre: Skipped 2 previous similar messages [ 4044.768919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4044.838673] Lustre: server umount lustre-MDT0000 complete [ 4046.124780] LustreError: 278108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777597060 with bad export cookie 17235253680141155547 [ 4046.129267] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4046.130248] LustreError: 278108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4046.260764] Lustre: server umount lustre-MDT0001 complete [ 4057.098089] Lustre: server umount lustre-OST2710 complete [ 4068.640967] Lustre: server umount lustre-OST03e8 complete [ 4074.537456] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 4077.549033] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 4089.760841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4089.880507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4089.891596] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4089.937246] Lustre: lustre-MDT0000: new disk, initializing [ 4089.965812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4089.972537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4091.246727] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4095.327612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4095.367042] Lustre: 285835:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4095.483473] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4096.720091] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4099.667828] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4100.883704] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4103.155672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4103.254824] Lustre: lustre-OST0000: new disk, initializing [ 4103.256339] Lustre: Skipped 1 previous similar message [ 4103.257909] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4103.259983] Lustre: Skipped 2 previous similar messages [ 4103.284210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4103.286457] Lustre: Skipped 1 previous similar message [ 4104.314262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4104.316968] Lustre: Skipped 1 previous similar message [ 4104.318564] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4104.344909] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4105.094117] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4108.098824] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4109.414728] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4109.496064] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4110.794049] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4110.869568] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4114.913578] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4114.916089] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4114.917645] LustreError: Skipped 1 previous similar message [ 4114.918528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4114.927218] Lustre: Skipped 1 previous similar message [ 4114.929811] Lustre: Skipped 3 previous similar messages [ 4117.579211] Lustre: server umount lustre-OST0000 complete [ 4120.033662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4120.034316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4120.037237] Lustre: Skipped 1 previous similar message [ 4120.039036] Lustre: Skipped 1 previous similar message [ 4125.152211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4125.154133] Lustre: Skipped 2 previous similar messages [ 4133.343191] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4133.415905] Lustre: server umount lustre-MDT0000 complete [ 4134.595542] LustreError: 285827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777597148 with bad export cookie 17235253680141157213 [ 4134.598940] LustreError: 285827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4134.600170] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4134.727503] Lustre: server umount lustre-MDT0001 complete [ 4137.616231] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 20:59:11 (1777597151) [ 4138.138561] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4138.745659] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 20:59:12 (1777597152) [ 4139.714028] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 4149.464780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4149.574997] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4149.585544] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4149.661857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4151.062886] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4154.115682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4154.147059] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4154.156981] Lustre: 291264:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4155.653010] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4171.874526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4171.978512] Lustre: Denying initial registration attempt from nid 192.168.204.125@tcp, specified as failover [ 4171.981939] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4171.987918] LustreError: 292684:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 4171.993520] LustreError: 292684:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 4171.997259] LustreError: 292684:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4172.027976] Lustre: server umount lustre-OST0000 complete [ 4172.029156] LustreError: 292684:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4174.816119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4174.820402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4174.827686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4174.830868] Lustre: Skipped 2 previous similar messages [ 4180.819062] Lustre: server umount lustre-MDT0000 complete [ 4182.415332] LustreError: 291257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777597196 with bad export cookie 17235253680141158081 [ 4182.421795] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4182.422935] LustreError: 291257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4182.550923] Lustre: server umount lustre-MDT0001 complete [ 4186.238190] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 4187.536540] Key type lgssc unregistered [ 4187.674643] LNet: 293787:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4187.680186] LNetError: 293787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4187.689737] LNet: Removed LNI 192.168.204.125@tcp [ 4188.080494] Key type .llcrypt unregistered [ 4188.081844] Key type ._llcrypt unregistered [ 4191.871842] Key type ._llcrypt registered [ 4191.872901] Key type .llcrypt registered [ 4191.920393] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 4192.518413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4192.527771] alg: No test for adler32 (adler32-zlib) [ 4193.377792] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 4193.460816] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 4195.039179] Key type lgssc registered [ 4195.407163] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4220.087937] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 4223.998770] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4224.009634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4225.128911] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4225.139371] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4225.210081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4227.011674] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4230.517622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4230.554155] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4230.565496] Lustre: 296492:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4230.671930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4232.026778] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4233.857330] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4235.131696] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4250.532710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4250.657297] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4250.716429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4252.701230] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4254.832130] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4255.636596] Lustre: server umount lustre-OST0000 complete [ 4258.290455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4258.292127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4258.295735] Lustre: Skipped 1 previous similar message [ 4263.349293] Lustre: server umount lustre-MDT0000 complete [ 4264.951573] LustreError: 296483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777597279 with bad export cookie 11782047301763108510 [ 4264.954787] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4264.958868] LustreError: 296483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4265.108510] Lustre: server umount lustre-MDT0001 complete [ 4269.035268] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 4270.463677] Key type lgssc unregistered [ 4270.632724] LNet: 299710:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4270.637895] LNetError: 299710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4270.649862] LNet: Removed LNI 192.168.204.125@tcp [ 4271.186255] Key type .llcrypt unregistered [ 4271.189165] Key type ._llcrypt unregistered [ 4280.434978] Key type ._llcrypt registered [ 4280.436107] Key type .llcrypt registered [ 4280.474115] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 4280.884126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4280.940272] alg: No test for adler32 (adler32-zlib) [ 4281.812352] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 4281.928700] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 4283.519236] Key type lgssc registered [ 4284.163256] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4287.123983] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4287.135706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4288.348871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4289.977650] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4293.080889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4293.219838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4294.966137] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4297.587261] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4298.903500] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4303.329959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4303.486909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4305.587862] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4307.611168] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4320.296112] LustreError: 303349:0:(ldlm_resource.c:1180:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x200001b71:0x1:0x0].0x85ae478e (00000000f3daac44) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4323.219977] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.25@tcp (stopping) [ 4323.302625] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4323.310240] Lustre: Skipped 2 previous similar messages [ 4326.464214] Lustre: server umount lustre-MDT0000 complete [ 4327.892702] LustreError: 301066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777597342 with bad export cookie 6423558536278364043 [ 4327.896291] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4327.898079] LustreError: 301066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4328.329205] LustreError: 301081:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4328.344893] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.25@tcp (stopping) [ 4328.350294] LustreError: 301081:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4328.357545] Lustre: Skipped 3 previous similar messages [ 4328.417490] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4332.448577] LustreError: 303238:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4332.449112] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4332.458094] LustreError: 303238:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4332.461615] Lustre: Skipped 1 previous similar message [ 4337.632100] LustreError: 301083:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4337.632409] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4337.637107] LustreError: 301083:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4337.638986] Lustre: Skipped 1 previous similar message [ 4342.239174] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4342.349549] Lustre: server umount lustre-MDT0001 complete [ 4345.318650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4349.208184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4352.992180] LustreError: 300396:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007032cf6b x1863945862274432/t0(0) o250->MGC192.168.204.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4353.173825] LustreError: 302689:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4353.184178] LustreError: 302689:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4353.216395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4353.254767] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4354.903457] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4358.175631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4358.306840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4359.599847] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4361.300657] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4368.872773] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4370.141318] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 8 sec [ 4371.621838] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4372.231187] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4379.105597] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4379.106993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4379.111360] LustreError: Skipped 1 previous similar message [ 4379.121202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4379.127790] Lustre: Skipped 3 previous similar messages [ 4388.319134] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4388.379546] Lustre: server umount lustre-OST0000 complete [ 4394.466794] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4394.467123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4394.468198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4394.468207] Lustre: Skipped 2 previous similar messages [ 4394.486254] Lustre: Skipped 2 previous similar messages [ 4404.191228] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4404.291607] Lustre: server umount lustre-MDT0000 complete [ 4404.705680] LustreError: 306056:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4404.716953] LustreError: 306056:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4405.887249] LustreError: 302703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777597420 with bad export cookie 6423558536278371162 [ 4405.891670] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4405.892859] LustreError: 302703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4406.020073] Lustre: server umount lustre-MDT0001 complete [ 4410.044631] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 4411.212165] Key type lgssc unregistered [ 4411.344532] LNet: 306790:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4411.349102] LNetError: 306790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4411.359677] LNet: Removed LNI 192.168.204.125@tcp [ 4411.771768] Key type .llcrypt unregistered [ 4411.772946] Key type ._llcrypt unregistered [ 4420.866970] Key type ._llcrypt registered [ 4420.868493] Key type .llcrypt registered [ 4420.910124] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 4421.331426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4421.371448] alg: No test for adler32 (adler32-zlib) [ 4422.236116] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 4422.325358] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 4423.911214] Key type lgssc registered [ 4424.421153] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4427.354328] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4427.363523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4428.558542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4430.036339] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4433.420651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4433.538231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4434.781850] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4436.401861] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4437.734824] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4443.234490] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4443.236534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4443.238666] Lustre: Skipped 1 previous similar message [ 4445.268840] Lustre: server umount lustre-MDT0000 complete [ 4446.554383] LustreError: 309559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777597460 with bad export cookie 17967761447846332933 [ 4446.558992] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4446.561341] LustreError: 309559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4446.697771] Lustre: server umount lustre-MDT0001 complete [ 4452.037840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4452.158307] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4452.168857] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4452.255973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4453.569889] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4456.861413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4456.895941] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4456.906255] Lustre: 310529:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4457.019582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4458.441980] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4460.463160] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4461.644666] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4466.660859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4466.788705] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4466.794070] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4466.800294] LustreError: 312185:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 4466.804252] LustreError: 312185:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 4466.808563] LustreError: 312185:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4466.851893] Lustre: server umount lustre-OST0000 complete [ 4466.855110] LustreError: 312185:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4472.232775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4472.332546] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4472.412116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4474.394663] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4476.617738] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4479.003263] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 4479.857768] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4479.914733] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4479.960318] Lustre: lustre-OST0001: new disk, initializing [ 4479.965534] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4482.477004] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4484.821948] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4485.132300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4485.136960] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4485.163174] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4490.209836] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4490.213729] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4490.215940] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4490.219714] Lustre: Skipped 1 previous similar message [ 4490.225080] Lustre: Skipped 2 previous similar messages [ 4492.163546] Lustre: server umount lustre-OST0001 complete [ 4494.304837] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4494.305089] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4494.310098] LustreError: Skipped 2 previous similar messages [ 4494.320829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4494.326340] Lustre: Skipped 1 previous similar message [ 4495.331509] LustreError: 312649:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4495.342617] LustreError: 312649:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 4500.452467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4500.454774] LustreError: 314171:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4500.456803] Lustre: Skipped 3 previous similar messages [ 4500.469987] LustreError: 314171:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4505.573641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4505.574074] LustreError: 314503:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4505.578172] Lustre: Skipped 1 previous similar message [ 4505.589407] LustreError: 314503:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4508.127126] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4508.203592] Lustre: server umount lustre-OST0000 complete [ 4510.690478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4510.699531] Lustre: Skipped 2 previous similar messages [ 4515.808201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4515.810813] Lustre: Skipped 3 previous similar messages [ 4516.062449] Lustre: server umount lustre-MDT0000 complete [ 4517.337403] LustreError: 312676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777597531 with bad export cookie 17967761447846333472 [ 4517.340174] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4517.342709] LustreError: 312676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4517.461773] Lustre: server umount lustre-MDT0001 complete [ 4521.523484] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 4522.734883] Key type lgssc unregistered [ 4522.861486] LNet: 315440:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4522.865363] LNetError: 315440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4522.874464] LNet: Removed LNI 192.168.204.125@tcp [ 4523.317566] Key type .llcrypt unregistered [ 4523.320044] Key type ._llcrypt unregistered [ 4555.928449] Key type ._llcrypt registered [ 4555.929806] Key type .llcrypt registered [ 4555.974573] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 4561.910276] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 4562.367457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4562.416243] alg: No test for adler32 (adler32-zlib) [ 4563.275856] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 4563.360966] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 4564.943171] Key type lgssc registered [ 4565.309056] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4576.770798] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4576.776912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4577.877310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4577.888499] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4577.926885] Lustre: lustre-MDT0000: new disk, initializing [ 4577.951911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4577.959148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4579.231647] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4584.053562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4584.122224] Lustre: 320310:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4584.157501] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4584.160754] Lustre: Skipped 1 previous similar message [ 4584.202227] Lustre: lustre-MDT0001: new disk, initializing [ 4584.236460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4584.248910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4584.253047] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4585.586843] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4588.825488] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4590.221829] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4593.145349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4593.251440] Lustre: lustre-OST0000: new disk, initializing [ 4593.254130] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4593.285485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4593.927369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4593.934057] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4593.985801] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4595.432659] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4598.913287] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4600.587749] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4600.671033] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4602.086745] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4602.194699] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4604.387912] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4604.388281] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4604.390017] LustreError: Skipped 1 previous similar message [ 4604.390555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4604.396164] Lustre: Skipped 1 previous similar message [ 4609.109141] Lustre: server umount lustre-OST0000 complete [ 4614.624708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4614.625225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4614.628921] Lustre: Skipped 1 previous similar message [ 4614.630679] Lustre: Skipped 1 previous similar message [ 4616.488862] Lustre: server umount lustre-MDT0000 complete [ 4617.764192] LustreError: 320301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777597631 with bad export cookie 16628042899590971928 [ 4617.766658] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4617.768089] LustreError: 320301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4617.876200] Lustre: server umount lustre-MDT0001 complete [ 4620.660882] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 21:07:14 (1777597634) [ 4623.493638] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 21:07:17 (1777597637) [ 4626.926200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4627.156079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4628.427826] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4631.388174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4631.530615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4632.920629] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4634.807318] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4636.029595] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4638.747140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4640.957245] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4642.756074] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4656.786670] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 4657.330243] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4657.858332] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4658.400178] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 4659.054971] Lustre: Failing over lustre-MDT0000 [ 4659.212348] Lustre: server umount lustre-MDT0000 complete [ 4662.239776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4662.242654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4662.247307] LustreError: 325362:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4662.253593] LustreError: 325362:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4662.753850] LustreError: 326801:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4662.761691] LustreError: 326801:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4667.872222] LustreError: 324652:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4667.877215] LustreError: 324652:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 4671.985692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4672.026416] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4672.116092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4672.118100] Lustre: Skipped 1 previous similar message [ 4672.136537] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4673.432799] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4673.927182] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4677.606784] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4677.614926] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4677.634964] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4678.778593] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4679.293282] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4681.122575] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 4681.759549] Lustre: Failing over lustre-MDT0000 [ 4681.988920] Lustre: server umount lustre-MDT0000 complete [ 4682.721052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4682.721189] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4682.722191] LustreError: 324646:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4682.722200] LustreError: 324646:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4682.725166] Lustre: Skipped 3 previous similar messages [ 4687.840180] LustreError: 325362:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4687.847779] LustreError: 325362:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 4694.970157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4695.015630] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4695.109457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4695.132376] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4696.356881] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4699.527727] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4700.138975] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4700.141092] Lustre: Skipped 2 previous similar messages [ 4700.149142] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4700.167169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 4701.295544] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4701.824938] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4705.251594] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4705.254193] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4705.254687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4705.254698] Lustre: Skipped 1 previous similar message [ 4705.257922] Lustre: Skipped 3 previous similar messages [ 4710.169044] Lustre: server umount lustre-MDT0000 complete [ 4710.369033] LustreError: 325362:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4710.373859] LustreError: 325362:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 7 previous similar messages [ 4711.365979] LustreError: 324676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777597725 with bad export cookie 16628042899590974581 [ 4711.368795] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4711.370312] LustreError: 324676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4711.487318] Lustre: server umount lustre-MDT0001 complete [ 4721.945751] Lustre: server umount lustre-OST0000 complete [ 4726.930157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4727.083084] LustreError: 330435:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4728.423618] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4731.187490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4731.301925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4731.304276] Lustre: Skipped 1 previous similar message [ 4732.562784] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4734.357388] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4735.609327] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4737.833184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4739.743739] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4741.418381] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4752.293716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 4761.567823] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4761.570434] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4761.574977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4761.578029] Lustre: Skipped 2 previous similar messages [ 4764.236735] Lustre: server umount lustre-OST0000 complete [ 4767.200285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4767.202195] Lustre: Skipped 2 previous similar messages [ 4771.673205] Lustre: server umount lustre-MDT0000 complete [ 4772.832323] LustreError: 331145:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4772.837145] LustreError: 331145:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4772.995253] LustreError: 330416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777597787 with bad export cookie 16628042899590975323 [ 4772.998718] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4772.999450] LustreError: 330416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4773.135652] Lustre: server umount lustre-MDT0001 complete [ 4776.873547] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 4777.922525] Key type lgssc unregistered [ 4778.062459] LNet: 333629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4778.066303] LNetError: 333629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4778.077294] LNet: Removed LNI 192.168.204.125@tcp [ 4778.383233] Key type .llcrypt unregistered [ 4778.384659] Key type ._llcrypt unregistered [ 4787.971278] Key type ._llcrypt registered [ 4787.972519] Key type .llcrypt registered [ 4788.009966] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 4788.446941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4788.460134] alg: No test for adler32 (adler32-zlib) [ 4789.331663] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 4789.419599] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 4791.007152] Key type lgssc registered [ 4791.390525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4793.953771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4793.958982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4795.125903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4796.434118] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4799.223323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4799.373985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4800.790776] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4802.560105] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4803.738941] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4805.949827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4806.063186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4807.936122] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4809.701129] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4820.394055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 4830.176097] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4830.179850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4830.185914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4830.689366] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4830.690367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4830.692615] Lustre: Skipped 1 previous similar message [ 4835.428612] Lustre: server umount lustre-MDT0000 complete [ 4835.809197] LustreError: 337262:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4835.815806] LustreError: 337262:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 4836.655168] LustreError: 335086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777597850 with bad export cookie 2151223082687745183 [ 4836.658389] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4836.658707] LustreError: 335086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4836.773646] Lustre: server umount lustre-MDT0001 complete [ 4848.352860] Lustre: server umount lustre-OST0000 complete [ 4852.790903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4852.943979] LustreError: 338494:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4852.967227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4854.221356] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4857.010053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4857.134316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4858.425823] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4860.280128] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4861.557432] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4863.849064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4865.857476] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4867.639235] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4878.315946] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4893.152046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4893.154314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4893.158132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4893.161062] Lustre: Skipped 1 previous similar message [ 4896.164473] Lustre: server umount lustre-MDT0000 complete [ 4897.445077] LustreError: 338474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777597911 with bad export cookie 2151223082687746282 [ 4897.447150] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4897.448689] LustreError: 338474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4897.569135] Lustre: server umount lustre-MDT0001 complete [ 4908.071776] Lustre: server umount lustre-OST0000 complete [ 4913.391349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4913.580633] LustreError: 342084:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4913.604637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4913.606863] Lustre: Skipped 1 previous similar message [ 4914.861170] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4917.652752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4919.084871] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4920.838708] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4922.060248] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4924.607769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4926.647910] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4928.385912] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4938.091855] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4943.331852] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4943.333177] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4943.338555] Lustre: Skipped 2 previous similar messages [ 4943.345278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4943.352164] Lustre: Skipped 3 previous similar messages [ 4946.378751] Lustre: server umount lustre-OST0000 complete [ 4948.448952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4948.449620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4948.450455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4948.451307] LustreError: Skipped 1 previous similar message [ 4948.458894] Lustre: Skipped 1 previous similar message [ 4954.235517] Lustre: server umount lustre-MDT0000 complete [ 4955.551511] LustreError: 342066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777597969 with bad export cookie 2151223082687747290 [ 4955.554947] LustreError: 342066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4955.556080] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4955.678672] Lustre: server umount lustre-MDT0001 complete [ 4959.449081] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 4960.501964] Key type lgssc unregistered [ 4960.636431] LNet: 345278:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4960.638826] LNetError: 345278:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4960.648382] LNet: Removed LNI 192.168.204.125@tcp [ 4960.967538] Key type .llcrypt unregistered [ 4960.969037] Key type ._llcrypt unregistered [ 4971.289266] Key type ._llcrypt registered [ 4971.290545] Key type .llcrypt registered [ 4971.324953] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 4971.727913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4971.739428] alg: No test for adler32 (adler32-zlib) [ 4972.613164] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 4972.708872] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 4974.287232] Key type lgssc registered [ 4974.689411] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4977.461254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4977.466918] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4977.469634] LustreError: 346677:0:(osd_handler.c:8530:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 4977.477896] LustreError: 346677:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 4977.480175] LustreError: 346677:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 4977.482412] LustreError: 346677:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 4977.485217] LustreError: 346677:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4980.323271] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4980.328917] LustreError: 347075:0:(osd_handler.c:8530:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 4980.340145] LustreError: 347075:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 4980.346821] LustreError: 347075:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 4980.351900] LustreError: 347075:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 4980.357528] LustreError: 347075:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4986.546922] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 4987.575086] Key type lgssc unregistered [ 4987.715391] LNet: 348058:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4987.717808] LNetError: 348058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4987.729284] LNet: Removed LNI 192.168.204.125@tcp [ 4988.056539] Key type .llcrypt unregistered [ 4988.057879] Key type ._llcrypt unregistered [ 4993.893486] Key type ._llcrypt registered [ 4993.894732] Key type .llcrypt registered [ 4993.934481] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 5000.011371] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 5000.372273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5000.432424] alg: No test for adler32 (adler32-zlib) [ 5001.279765] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 5001.358888] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 5002.935153] Key type lgssc registered [ 5003.297433] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5013.914268] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5013.920488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5015.006367] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5015.017558] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5015.059457] Lustre: lustre-MDT0000: new disk, initializing [ 5015.088460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5015.095490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5016.425942] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5020.407089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5020.448850] Lustre: 351153:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5020.473231] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5020.475348] Lustre: Skipped 1 previous similar message [ 5020.516042] Lustre: lustre-MDT0001: new disk, initializing [ 5020.547459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5020.558558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5020.562189] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5021.796933] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5024.783480] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5025.918525] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5028.059648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5028.157045] Lustre: lustre-OST0000: new disk, initializing [ 5028.158835] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5028.185677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5030.004871] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5032.916734] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5034.186481] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5034.492937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5034.496402] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5034.535623] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5035.278983] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5036.700731] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5036.789872] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5039.586956] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5039.587708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5039.590850] Lustre: Skipped 1 previous similar message [ 5039.593867] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5043.603292] Lustre: server umount lustre-OST0000 complete [ 5046.239994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5046.242217] LustreError: Skipped 1 previous similar message [ 5046.243450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5046.247114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5046.249839] Lustre: Skipped 1 previous similar message [ 5049.824393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5049.824796] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5054.944216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5054.946024] Lustre: Skipped 2 previous similar messages [ 5059.039137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5059.040362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5059.045811] Lustre: Skipped 1 previous similar message [ 5059.111540] Lustre: server umount lustre-MDT0000 complete [ 5060.310240] LustreError: 351143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777598074 with bad export cookie 12892341775277940052 [ 5060.313050] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5060.314697] LustreError: 351143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5060.436886] Lustre: server umount lustre-MDT0001 complete [ 5063.050551] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5063.582182] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 21:14:37 (1777598077) [ 5065.723038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5065.931404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5067.206940] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5070.168878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5070.297260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5071.528286] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5073.398061] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5074.511910] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5076.652139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5078.684389] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5080.419414] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5082.584752] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5082.650611] Lustre: lustre-OST0001: new disk, initializing [ 5082.652440] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5082.683442] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5082.685377] Lustre: Skipped 1 previous similar message [ 5084.159988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5084.162816] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5084.196377] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5084.457432] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5087.528884] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5091.426705] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5091.429098] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5091.431260] Lustre: Skipped 1 previous similar message [ 5095.487403] Lustre: server umount lustre-OST0001 complete [ 5096.934354] LustreError: 356804:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5096.942418] LustreError: 356804:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 5102.051842] LustreError: 356805:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5102.064237] LustreError: 356805:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5107.175126] LustreError: 357979:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5107.183391] LustreError: 357979:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5111.776465] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5111.781937] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5111.790931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5111.794789] Lustre: Skipped 2 previous similar messages [ 5112.292170] LustreError: 356804:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5112.300815] LustreError: 356804:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5114.008004] Lustre: server umount lustre-OST0000 complete [ 5116.384396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5116.390043] LustreError: Skipped 1 previous similar message [ 5121.761161] Lustre: server umount lustre-MDT0000 complete [ 5122.528145] LustreError: 355230:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5122.532944] LustreError: 355230:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5122.977147] LustreError: 355928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777598137 with bad export cookie 12892341775277940920 [ 5122.981146] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5122.982701] LustreError: 355928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5137.375126] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5137.486368] Lustre: server umount lustre-MDT0001 complete [ 5141.675769] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 5143.227438] Key type lgssc unregistered [ 5143.413558] LNet: 359557:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5143.419245] LNetError: 359557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5143.428637] LNet: Removed LNI 192.168.204.125@tcp [ 5143.909142] Key type .llcrypt unregistered [ 5143.911623] Key type ._llcrypt unregistered [ 5162.350939] Key type ._llcrypt registered [ 5162.352684] Key type .llcrypt registered [ 5162.398022] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 5162.849798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5162.939737] alg: No test for adler32 (adler32-zlib) [ 5163.820122] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 5163.911031] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 5165.503138] Key type lgssc registered [ 5165.967236] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5168.803100] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5168.810843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5169.927455] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5169.935871] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5170.007152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5171.294613] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5174.188385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5174.223231] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5174.237525] Lustre: 362315:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5174.339840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5175.712637] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5177.908513] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5179.161995] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5181.409396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5181.410847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5181.418510] Lustre: Skipped 1 previous similar message [ 5181.425586] Lustre: Skipped 1 previous similar message [ 5186.529226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5186.533537] Lustre: Skipped 1 previous similar message [ 5191.649499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5191.656712] Lustre: Skipped 1 previous similar message [ 5194.207148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5194.301849] Lustre: server umount lustre-MDT0000 complete [ 5195.948195] LustreError: 362307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777598210 with bad export cookie 6461931543503992547 [ 5195.954405] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5195.957153] LustreError: 362307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5196.120362] Lustre: server umount lustre-MDT0001 complete [ 5199.499512] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 21:16:53 (1777598213) [ 5202.000620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5202.171285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5203.451184] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5206.254170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5206.384476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5207.668550] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5209.346300] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5210.462691] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5212.784022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5212.849726] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5212.921697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5214.979893] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5216.876919] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5220.567237] Lustre: Permanently deactivating lustre-OST0000 [ 5220.573070] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5222.383662] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5223.743589] LustreError: 367149:0:(mgs_llog.c:2031:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5223.747453] LustreError: 367149:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 5223.749284] LustreError: 367149:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5224.338247] LustreError: 367197:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 5224.340988] LustreError: 367197:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5231.116032] Lustre: server umount lustre-OST0000 complete [ 5232.037734] LustreError: 367446:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 5232.040301] LustreError: 367446:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5237.216786] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5237.216795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5237.217397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5237.220582] Lustre: Skipped 1 previous similar message [ 5242.336075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5242.338234] Lustre: Skipped 2 previous similar messages [ 5242.361956] Lustre: server umount lustre-MDT0000 complete [ 5243.572433] LustreError: 366206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777598257 with bad export cookie 6461931543503993114 [ 5243.577121] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5243.577452] LustreError: 366206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5253.134946] Lustre: server umount lustre-MDT0001 complete [ 5255.907218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5257.212682] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5258.092821] LustreError: 368584:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5260.213478] Lustre: 368923:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.125@tcp [ 5260.888300] Lustre: server umount MGS complete [ 5264.513239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5264.683719] LustreError: 369646:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5264.688536] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5264.688814] LustreError: 369646:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5264.690972] Lustre: Skipped 1 previous similar message [ 5264.711543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5266.010282] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5268.968916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5269.071453] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5270.433271] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5272.159353] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5273.362260] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5273.984187] Lustre: Permanently reactivating lustre-OST0000 [ 5273.989385] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5273.991571] Lustre: Skipped 2 previous similar messages [ 5300.191144] LustreError: 371249:0:(osp_dev.c:695:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5302.942597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5303.073422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5303.075358] Lustre: Skipped 1 previous similar message [ 5304.957209] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5306.697039] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5319.648148] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5319.650092] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5319.653777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5320.679622] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5324.377920] Lustre: server umount lustre-OST0000 complete [ 5325.792087] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5325.795670] Lustre: Skipped 1 previous similar message [ 5330.399827] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5330.404291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5331.790629] Lustre: server umount lustre-MDT0000 complete [ 5332.913982] LustreError: 369627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777598347 with bad export cookie 6461931543503994136 [ 5332.917367] LustreError: 369627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5332.918036] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5333.034135] Lustre: server umount lustre-MDT0001 complete [ 5336.515963] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 5337.592682] Key type lgssc unregistered [ 5337.714540] LNet: 373958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5337.719461] LNetError: 373958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5337.730407] LNet: Removed LNI 192.168.204.125@tcp [ 5338.093934] Key type .llcrypt unregistered [ 5338.095281] Key type ._llcrypt unregistered [ 5344.232152] Key type ._llcrypt registered [ 5344.233830] Key type .llcrypt registered [ 5344.270508] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 5350.301911] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 5350.626931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5350.656463] alg: No test for adler32 (adler32-zlib) [ 5351.506149] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 5351.592862] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 5353.175236] Key type lgssc registered [ 5353.629843] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5367.929676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5367.935525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5369.041304] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5369.054869] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5369.093972] Lustre: lustre-MDT0000: new disk, initializing [ 5369.118195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5369.125469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5370.473978] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5375.548403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5375.620119] Lustre: 377053:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5375.661890] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5375.665889] Lustre: Skipped 1 previous similar message [ 5375.715928] Lustre: lustre-MDT0001: new disk, initializing [ 5375.751960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5375.764945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5375.768507] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5377.459533] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5381.346301] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5382.948536] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5385.445755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5385.579363] Lustre: lustre-OST0000: new disk, initializing [ 5385.581230] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5385.612772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5385.981540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5385.986069] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5386.037747] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5387.720431] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5391.492070] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5393.043470] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5393.115908] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5394.453755] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5394.567852] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5396.450804] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5396.453750] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5396.455820] LustreError: Skipped 1 previous similar message [ 5396.465558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5396.469757] Lustre: Skipped 1 previous similar message [ 5401.572453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5401.576516] Lustre: Skipped 1 previous similar message [ 5401.645705] Lustre: server umount lustre-OST0000 complete [ 5406.689561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5406.692070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5406.702074] Lustre: Skipped 1 previous similar message [ 5406.705692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5406.709397] Lustre: Skipped 1 previous similar message [ 5409.694945] Lustre: server umount lustre-MDT0000 complete [ 5411.369313] LustreError: 377831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777598425 with bad export cookie 8134685581825096819 [ 5411.376614] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5411.379891] LustreError: 377831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5411.539252] Lustre: server umount lustre-MDT0001 complete [ 5415.760376] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5416.342697] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 21:20:30 (1777598430) [ 5419.359163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5419.624472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5421.449513] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5424.875792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5425.036879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5426.516030] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5428.400572] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5429.547554] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5432.322296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5434.272659] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5435.833342] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5436.401674] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5447.128683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5450.210009] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5450.211279] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5450.212273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5450.220232] Lustre: Skipped 2 previous similar messages [ 5454.802109] Lustre: server umount lustre-OST0000 complete [ 5460.449170] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5460.449774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5460.456782] Lustre: Skipped 1 previous similar message [ 5460.459096] Lustre: Skipped 1 previous similar message [ 5462.646107] Lustre: server umount lustre-MDT0000 complete [ 5464.417571] LustreError: 381131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777598478 with bad export cookie 8134685581825097687 [ 5464.422214] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5464.426942] LustreError: 381131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5464.553953] Lustre: server umount lustre-MDT0001 complete [ 5468.880167] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 5470.176436] Key type lgssc unregistered [ 5470.330587] LNet: 384491:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5470.335461] LNetError: 384491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5470.343582] LNet: Removed LNI 192.168.204.125@tcp [ 5470.780265] Key type .llcrypt unregistered [ 5470.781771] Key type ._llcrypt unregistered [ 5480.090281] Key type ._llcrypt registered [ 5480.091652] Key type .llcrypt registered [ 5480.131717] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 5480.235340] Key type .llcrypt unregistered [ 5480.236614] Key type ._llcrypt unregistered [ 5488.079598] Key type ._llcrypt registered [ 5488.081503] Key type .llcrypt registered [ 5488.129981] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 5488.560292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5488.570858] alg: No test for adler32 (adler32-zlib) [ 5489.440195] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 5489.535450] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 5491.119270] Key type lgssc registered [ 5491.516242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5494.436461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5494.449950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5495.637471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5497.102562] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5500.721171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5500.858253] LustreError: 386642:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5500.862618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5500.867131] LustreError: 386642:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5503.214327] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5504.994373] LustreError: 386645:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5505.000448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5505.005028] LustreError: 386645:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5505.656950] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5509.118284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5509.306776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5510.980270] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5519.843970] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5519.847994] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5519.852370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5519.860362] Lustre: Skipped 1 previous similar message [ 5524.906330] Lustre: server umount lustre-OST0000 complete [ 5529.059193] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5529.064483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5529.072396] Lustre: Skipped 1 previous similar message [ 5529.075974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5530.080295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5530.080919] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5532.916034] Lustre: server umount lustre-MDT0000 complete [ 5534.542754] LustreError: 386626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777598548 with bad export cookie 14128896285337143538 [ 5534.549504] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5534.553075] LustreError: 386626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5534.707157] Lustre: server umount lustre-MDT0001 complete [ 5539.502705] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 5540.700274] Key type lgssc unregistered [ 5540.826520] LNet: 389627:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5540.832376] LNetError: 389627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5540.850440] LNet: Removed LNI 192.168.204.125@tcp [ 5541.189411] Key type .llcrypt unregistered [ 5541.191929] Key type ._llcrypt unregistered [ 5551.290418] Key type ._llcrypt registered [ 5551.292326] Key type .llcrypt registered [ 5551.348309] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 5551.779233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5551.789848] alg: No test for adler32 (adler32-zlib) [ 5552.660215] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 5552.753675] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 5554.359211] Key type lgssc registered [ 5554.884665] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5558.392345] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5558.402226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5573.855353] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5578.975231] LustreError: 390983:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.125@tcp: failed processing log, type 4: rc = -110 [ 5605.599143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5607.698402] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5609.729271] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5612.412389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5612.634325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5612.674304] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5613.941950] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5617.260263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5617.393612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5618.717879] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5620.596618] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5622.254317] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5625.827299] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5625.829085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5625.833424] Lustre: Skipped 1 previous similar message [ 5630.815421] Lustre: server umount lustre-OST0000 complete [ 5632.992406] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5632.997059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5633.008452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5633.012446] Lustre: Skipped 1 previous similar message [ 5636.065626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5636.065945] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5638.633140] Lustre: server umount lustre-MDT0000 complete [ 5640.028056] LustreError: 393172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777598654 with bad export cookie 15399349772457170621 [ 5640.031605] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5640.031744] LustreError: 393172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5640.134534] Lustre: server umount lustre-MDT0001 complete [ 5643.560173] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 5644.582087] Key type lgssc unregistered [ 5644.707596] LNet: 394192:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5644.710884] LNetError: 394192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5644.720394] LNet: Removed LNI 192.168.204.125@tcp [ 5645.038058] Key type .llcrypt unregistered [ 5645.039570] Key type ._llcrypt unregistered [ 5654.580486] Key type ._llcrypt registered [ 5654.581801] Key type .llcrypt registered [ 5654.616037] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 5654.893937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5654.900527] alg: No test for adler32 (adler32-zlib) [ 5655.748832] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 5655.827297] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 5657.415149] Key type lgssc registered [ 5657.789668] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5660.448497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5660.454527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5661.627850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5663.110779] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5666.178163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5666.322822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5667.530170] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5669.151298] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5670.210970] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5672.278027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5672.389117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5673.450359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5674.174328] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5675.909076] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5681.635115] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5681.635532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5681.636503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5681.646665] Lustre: Skipped 1 previous similar message [ 5683.857242] Lustre: server umount lustre-MDT0000 complete [ 5684.556493] LustreError: 395551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.25@tcp arrived at 1777598698 with bad export cookie 6068153391679677307 [ 5684.563192] LustreError: 395551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5685.140974] LustreError: 396285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777598699 with bad export cookie 6068153391679676460 [ 5685.144931] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5686.752923] LustreError: 395568:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5686.753109] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5686.761964] LustreError: 395568:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 5686.776057] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5686.786793] Lustre: Skipped 1 previous similar message [ 5686.790614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5686.794925] Lustre: Skipped 3 previous similar messages [ 5690.783840] LustreError: 396291:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5690.784116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5690.789263] LustreError: 396291:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5691.274091] Lustre: server umount lustre-OST0000 complete [ 5693.498179] Lustre: server umount lustre-MDT0001 complete [ 5697.541753] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 5698.739505] Key type lgssc unregistered [ 5698.911812] LNet: 398894:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5698.917523] LNetError: 398894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5698.926590] LNet: Removed LNI 192.168.204.125@tcp [ 5699.363891] Key type .llcrypt unregistered [ 5699.366628] Key type ._llcrypt unregistered [ 5709.156083] Key type ._llcrypt registered [ 5709.157233] Key type .llcrypt registered [ 5709.197431] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 5709.588832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5709.630753] alg: No test for adler32 (adler32-zlib) [ 5710.474882] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 5710.556996] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 5712.143157] Key type lgssc registered [ 5712.548437] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5715.216985] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5715.223346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5716.406589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5717.680463] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5720.271885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5720.392365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5721.570022] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5723.236220] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5724.361319] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5726.481328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5726.607577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5727.655574] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5728.354716] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5730.001780] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5735.905925] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5735.906551] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5735.912854] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5735.914563] Lustre: Skipped 1 previous similar message [ 5737.939314] Lustre: server umount lustre-MDT0001 complete [ 5741.025069] LustreError: 400272:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5741.025539] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5741.029757] LustreError: 400272:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5741.033189] Lustre: Skipped 1 previous similar message [ 5741.033647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5741.039272] Lustre: Skipped 1 previous similar message [ 5745.548597] Lustre: server umount lustre-OST0000 complete [ 5746.143946] LustreError: 401716:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5746.731153] Lustre: server umount lustre-MDT0000 complete [ 5750.925625] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 5751.810665] Key type lgssc unregistered [ 5751.927350] LNet: 403595:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5751.929676] LNetError: 403595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5751.940310] LNet: Removed LNI 192.168.204.125@tcp [ 5752.215450] Key type .llcrypt unregistered [ 5752.216537] Key type ._llcrypt unregistered [ 5759.752596] Key type ._llcrypt registered [ 5759.753711] Key type .llcrypt registered [ 5759.786184] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 5765.723424] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 5766.075823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5766.109718] alg: No test for adler32 (adler32-zlib) [ 5766.950580] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 5767.030209] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 5768.607143] Key type lgssc registered [ 5768.944302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5782.361731] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 5787.708047] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5787.713136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5788.800141] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5788.811334] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5788.855441] Lustre: lustre-MDT0000: new disk, initializing [ 5788.878251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5788.885898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5790.184923] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5795.854027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5795.948716] Lustre: lustre-OST0000: new disk, initializing [ 5795.950609] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5795.952268] Lustre: Skipped 1 previous similar message [ 5795.972196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5797.778908] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5800.887945] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5803.022864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 5803.028684] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 5803.042172] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 5811.876751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5811.921419] Lustre: 407531:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5811.951843] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5811.987935] Lustre: lustre-MDT0001: new disk, initializing [ 5812.037975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5812.053144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 5812.060513] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 5813.419669] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5816.961551] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5817.074991] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5818.393318] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5818.480776] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5822.945768] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5822.945938] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5822.949669] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5822.952910] Lustre: Skipped 1 previous similar message [ 5826.189391] Lustre: server umount lustre-MDT0001 complete [ 5833.185953] LustreError: 407542:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5833.191610] LustreError: 407542:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 5838.305073] LustreError: 407537:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5838.309509] LustreError: 407537:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5838.815974] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5838.818178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5838.821500] Lustre: Skipped 1 previous similar message [ 5838.823093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5838.824816] Lustre: Skipped 1 previous similar message [ 5843.424229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5843.424920] LustreError: 407542:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5844.496757] Lustre: server umount lustre-OST0000 complete [ 5851.962296] Lustre: server umount lustre-MDT0000 complete [ 5856.404166] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 5857.435600] Key type lgssc unregistered [ 5857.548299] LNet: 412097:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5857.550700] LNetError: 412097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5857.556360] LNet: Removed LNI 192.168.204.125@tcp [ 5857.805537] Key type .llcrypt unregistered [ 5857.807236] Key type ._llcrypt unregistered [ 5869.768429] Key type ._llcrypt registered [ 5869.769589] Key type .llcrypt registered [ 5869.807360] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 5869.902351] Key type .llcrypt unregistered [ 5869.903520] Key type ._llcrypt unregistered [ 5876.257743] Key type ._llcrypt registered [ 5876.258789] Key type .llcrypt registered [ 5876.290055] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 5876.580257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5876.607460] alg: No test for adler32 (adler32-zlib) [ 5877.444084] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 5877.515570] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 5879.087119] Key type lgssc registered [ 5879.405956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5888.040972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5888.051246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5889.170227] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5889.188778] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5889.236726] Lustre: lustre-MDT0000: new disk, initializing [ 5889.267183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5889.274392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5890.713935] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5894.816450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5894.854671] Lustre: 415412:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5894.890665] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5894.893987] Lustre: Skipped 1 previous similar message [ 5894.938024] Lustre: lustre-MDT0001: new disk, initializing [ 5894.970757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5894.989185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5894.995659] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5896.295223] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5899.303544] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5900.564634] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5902.766555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5902.866145] Lustre: lustre-OST0000: new disk, initializing [ 5902.867769] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5902.892117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5904.850859] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5907.998889] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5908.984118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5908.987425] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5909.016093] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5915.616682] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5915.619033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5915.622757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5919.201077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5919.201838] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5921.749554] Lustre: server umount lustre-MDT0000 complete [ 5922.903161] LustreError: 415403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777598937 with bad export cookie 8634787242558254480 [ 5922.906521] LustreError: 415403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5922.907272] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5923.017564] Lustre: server umount lustre-MDT0001 complete [ 5934.359475] Lustre: server umount lustre-OST0000 complete [ 5937.503191] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 21:29:11 (1777598951) [ 5939.930498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5940.108355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5941.266062] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5943.895867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5945.092343] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5946.647964] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5947.683165] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5962.977710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5963.040297] Lustre: Found index 0 for lustre-OST0000, updating log [ 5963.096386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5963.098424] Lustre: Skipped 1 previous similar message [ 5964.761147] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5965.157524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 5966.339643] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5969.441768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5969.441898] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5969.443565] Lustre: Skipped 2 previous similar messages [ 5969.448097] Lustre: Skipped 1 previous similar message [ 5973.446259] Lustre: server umount lustre-OST0000 complete [ 5989.984592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5989.988566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5989.990194] Lustre: Skipped 2 previous similar messages [ 5992.019943] Lustre: server umount lustre-MDT0000 complete [ 5993.185141] LustreError: 419047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777599007 with bad export cookie 8634787242558255873 [ 5993.188448] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5993.189049] LustreError: 419047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5993.312849] Lustre: server umount lustre-MDT0001 complete [ 5995.982091] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 21:30:10 (1777599010) [ 6027.039271] LNet: 1 local NIs in recovery (showing 1): 192.168.204.125@tcp [ 6581.484931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6581.710425] LustreError: 425125:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6581.719663] LustreError: 425125:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6581.749165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6583.438262] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6586.848563] LustreError: 425126:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6587.559311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6587.737424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6589.235837] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6592.025154] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6593.433845] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6609.571937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6609.743985] Lustre: Found index 0 for lustre-OST0000, updating log [ 6609.872238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6612.093527] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6613.961691] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6619.105298] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6619.106905] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6619.110626] Lustre: Skipped 1 previous similar message [ 6619.112746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6619.114447] Lustre: Skipped 2 previous similar messages [ 6620.820008] Lustre: server umount lustre-OST0000 complete [ 6623.457730] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6623.458447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6623.461555] Lustre: Skipped 1 previous similar message [ 6624.480948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6624.482798] Lustre: Skipped 2 previous similar messages [ 6626.529443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6626.532691] Lustre: Skipped 3 previous similar messages [ 6628.628502] Lustre: server umount lustre-MDT0000 complete [ 6629.601684] LustreError: 425836:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6629.605823] LustreError: 425836:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 6629.761891] LustreError: 425106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777599643 with bad export cookie 8634787242558256930 [ 6629.766818] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6629.767917] LustreError: 425106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6629.909354] Lustre: server umount lustre-MDT0001 complete [ 6640.415453] LNet: 1 local NIs in recovery (showing 1): 192.168.204.125@tcp [ 6648.550822] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 6652.267516] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 6666.766502] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 21:41:20 (1777599680) [ 6674.484973] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 6675.513298] Key type lgssc unregistered [ 6675.631855] LNet: 431387:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6675.635120] LNetError: 431387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6739.109595] LNet: Removed LNI 192.168.204.125@tcp [ 6739.455512] Key type .llcrypt unregistered [ 6739.457275] Key type ._llcrypt unregistered [ 6747.006351] Key type ._llcrypt registered [ 6747.007564] Key type .llcrypt registered [ 6747.040874] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 6747.329590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6747.377479] alg: No test for adler32 (adler32-zlib) [ 6748.235188] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 6748.314344] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 6749.895296] Key type lgssc registered [ 6750.443739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6752.243239] LNet: Added LNI 192.168.204.125@tcp100 [8/256/0/180] [ 6752.553846] LNet: Added LNI 192.168.204.125@tcp101 [8/256/0/180] [ 6752.956978] LNet: Added LNI 192.168.204.125@tcp102 [8/256/0/180] [ 6753.387187] LNet: Added LNI 192.168.204.125@tcp103 [8/256/0/180] [ 6753.808642] LNet: Added LNI 192.168.204.125@tcp104 [8/256/0/180] [ 6754.268121] LNet: Added LNI 192.168.204.125@tcp105 [8/256/0/180] [ 6754.699488] LNet: Added LNI 192.168.204.125@tcp106 [8/256/0/180] [ 6755.099452] LNet: Added LNI 192.168.204.125@tcp107 [8/256/0/180] [ 6755.520204] LNet: Added LNI 192.168.204.125@tcp108 [8/256/0/180] [ 6755.946662] LNet: Added LNI 192.168.204.125@tcp109 [8/256/0/180] [ 6756.388749] LNet: Added LNI 192.168.204.125@tcp110 [8/256/0/180] [ 6756.791290] LNet: Added LNI 192.168.204.125@tcp111 [8/256/0/180] [ 6757.054521] LNet: Added LNI 192.168.204.125@tcp112 [8/256/0/180] [ 6757.310974] LNet: Added LNI 192.168.204.125@tcp113 [8/256/0/180] [ 6757.564681] LNet: Added LNI 192.168.204.125@tcp114 [8/256/0/180] [ 6757.907730] LNet: Added LNI 192.168.204.125@tcp115 [8/256/0/180] [ 6758.282351] LNet: Added LNI 192.168.204.125@tcp116 [8/256/0/180] [ 6758.695528] LNet: Added LNI 192.168.204.125@tcp117 [8/256/0/180] [ 6759.066396] LNet: Added LNI 192.168.204.125@tcp118 [8/256/0/180] [ 6759.457177] LNet: Added LNI 192.168.204.125@tcp119 [8/256/0/180] [ 6759.872190] LNet: Added LNI 192.168.204.125@tcp120 [8/256/0/180] [ 6760.315276] LNet: Added LNI 192.168.204.125@tcp121 [8/256/0/180] [ 6760.663938] LNet: Added LNI 192.168.204.125@tcp122 [8/256/0/180] [ 6761.092025] LNet: Added LNI 192.168.204.125@tcp123 [8/256/0/180] [ 6761.525624] LNet: Added LNI 192.168.204.125@tcp124 [8/256/0/180] [ 6761.934249] LNet: Added LNI 192.168.204.125@tcp125 [8/256/0/180] [ 6762.205028] LNet: Added LNI 192.168.204.125@tcp126 [8/256/0/180] [ 6762.464603] LNet: Added LNI 192.168.204.125@tcp127 [8/256/0/180] [ 6762.733791] LNet: Added LNI 192.168.204.125@tcp128 [8/256/0/180] [ 6763.007275] LNet: Added LNI 192.168.204.125@tcp129 [8/256/0/180] [ 6763.411491] LNet: Added LNI 192.168.204.125@tcp130 [8/256/0/180] [ 6763.800564] LNet: Added LNI 192.168.204.125@tcp131 [8/256/0/180] [ 6764.236314] LNet: Added LNI 192.168.204.125@tcp132 [8/256/0/180] [ 6764.637218] LNet: Added LNI 192.168.204.125@tcp133 [8/256/0/180] [ 6765.069916] LNet: Added LNI 192.168.204.125@tcp134 [8/256/0/180] [ 6765.461251] LNet: Added LNI 192.168.204.125@tcp135 [8/256/0/180] [ 6768.262208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6768.272912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6769.336125] Lustre: 434366:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.125@tcp: can't add NID 192.168.204.125@tcp130: rc = -75 [ 6769.360235] LNetError: 431830:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.125 [ 6769.366468] LNetError: Protocol error connecting to 192.168.204.125@tcp119 at host 192.168.204.125:988: is it running a compatible version of Lustre? [ 6769.416449] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6769.431761] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6769.472626] Lustre: lustre-MDT0000: new disk, initializing [ 6769.505978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6769.514746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6769.823176] LNet: 1 local NIs in recovery (showing 1): 192.168.204.125@tcp119 [ 6770.403853] LNetError: 431830:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.125 [ 6770.411133] LNetError: Protocol error connecting to 192.168.204.125@tcp103 at host 192.168.204.125:988: is it running a compatible version of Lustre? [ 6771.164702] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6772.895961] LNetError: 434391:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.125 [ 6772.898669] LNetError: 434391:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Skipped 1 previous similar message [ 6772.900787] LNetError: Protocol error connecting to 192.168.204.125@tcp119 at host 192.168.204.125:988: is it running a compatible version of Lustre? [ 6772.903394] LNetError: Skipped 1 previous similar message [ 6776.580624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6776.651651] Lustre: 434382:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6776.691437] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6776.695452] Lustre: Skipped 1 previous similar message [ 6776.740213] Lustre: lustre-MDT0001: new disk, initializing [ 6776.754364] Lustre: 435165:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.125@tcp: can't add NID 192.168.204.125@tcp130: rc = -75 [ 6776.757125] Lustre: 435165:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 6776.812514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6776.832629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6776.838667] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6776.993148] LNetError: 431831:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.125 [ 6776.999118] LNetError: Protocol error connecting to 192.168.204.125@tcp119 at host 192.168.204.125:988: is it running a compatible version of Lustre? [ 6778.038001] Lustre: 435389:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.125@tcp: can't add NID 192.168.204.125@tcp130: rc = -75 [ 6778.043316] Lustre: 435389:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 6778.098815] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6782.054323] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6783.206835] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6786.365665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6786.550081] Lustre: lustre-OST0000: new disk, initializing [ 6786.553758] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6786.609588] Lustre: 436136:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.125@tcp: can't add NID 192.168.204.125@tcp130: rc = -75 [ 6786.615112] Lustre: 436136:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 6786.638501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6788.627680] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6792.601391] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6795.669379] Lustre: server umount lustre-OST0000 complete [ 6796.796921] Lustre: 437086:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.125@tcp: can't add NID 192.168.204.125@tcp130: rc = -75 [ 6796.802423] Lustre: 437086:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 6800.544715] LNetError: 434386:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.125 [ 6800.551717] LNetError: Protocol error connecting to 192.168.204.125@tcp119 at host 192.168.204.125:988: is it running a compatible version of Lustre? [ 6801.890243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6801.891362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6801.898178] Lustre: Skipped 1 previous similar message [ 6803.477025] Lustre: server umount lustre-MDT0000 complete [ 6804.843122] LustreError: 434374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777599818 with bad export cookie 15220444128380420418 [ 6804.845359] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6804.846539] LustreError: 434374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6804.945248] Lustre: server umount lustre-MDT0001 complete [ 6808.187505] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 6809.019378] Key type lgssc unregistered [ 6809.159938] LNet: 437907:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6809.162626] LNetError: 437907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6809.164535] LNet: Removed LNI 192.168.204.125@tcp [ 6810.207132] LNet: Removed LNI 192.168.204.125@tcp100 [ 6810.210651] LNetError: 437907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6810.214770] LNetError: 437907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 6810.219417] LNet: Removed LNI 192.168.204.125@tcp101 [ 6811.231122] LNet: Removed LNI 192.168.204.125@tcp102 [ 6811.232773] LNetError: 437907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6811.235510] LNetError: 437907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 6811.238993] LNet: Removed LNI 192.168.204.125@tcp103 [ 6812.255105] LNet: Removed LNI 192.168.204.125@tcp104 [ 6813.279127] LNet: Removed LNI 192.168.204.125@tcp105 [ 6813.280443] LNetError: 437907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6813.282542] LNetError: 437907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 6813.284612] LNet: Removed LNI 192.168.204.125@tcp106 [ 6814.303133] LNet: Removed LNI 192.168.204.125@tcp107 [ 6815.327101] LNet: Removed LNI 192.168.204.125@tcp108 [ 6816.351113] LNet: Removed LNI 192.168.204.125@tcp109 [ 6817.375170] LNet: Removed LNI 192.168.204.125@tcp110 [ 6817.377872] LNetError: 437907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6817.382418] LNetError: 437907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 6817.387652] LNet: Removed LNI 192.168.204.125@tcp111 [ 6818.399109] LNet: Removed LNI 192.168.204.125@tcp112 [ 6819.423123] LNet: Removed LNI 192.168.204.125@tcp113 [ 6820.447198] LNet: Removed LNI 192.168.204.125@tcp114 [ 6821.471196] LNet: Removed LNI 192.168.204.125@tcp115 [ 6822.495180] LNet: Removed LNI 192.168.204.125@tcp116 [ 6823.519179] LNet: Removed LNI 192.168.204.125@tcp117 [ 6824.543179] LNet: Removed LNI 192.168.204.125@tcp118 [ 6824.547200] LNet: Removed LNI 192.168.204.125@tcp119 [ 6824.550374] LNet: Removed LNI 192.168.204.125@tcp120 [ 6824.553293] LNet: Removed LNI 192.168.204.125@tcp121 [ 6824.556260] LNet: Removed LNI 192.168.204.125@tcp122 [ 6824.559172] LNet: Removed LNI 192.168.204.125@tcp123 [ 6824.562290] LNet: Removed LNI 192.168.204.125@tcp124 [ 6824.565580] LNet: Removed LNI 192.168.204.125@tcp125 [ 6824.568251] LNet: Removed LNI 192.168.204.125@tcp126 [ 6824.571074] LNet: Removed LNI 192.168.204.125@tcp127 [ 6824.573976] LNet: Removed LNI 192.168.204.125@tcp128 [ 6824.576990] LNet: Removed LNI 192.168.204.125@tcp129 [ 6824.580045] LNet: Removed LNI 192.168.204.125@tcp130 [ 6824.583044] LNet: Removed LNI 192.168.204.125@tcp131 [ 6824.585963] LNet: Removed LNI 192.168.204.125@tcp132 [ 6824.588881] LNet: Removed LNI 192.168.204.125@tcp133 [ 6824.591683] LNet: Removed LNI 192.168.204.125@tcp134 [ 6824.602810] LNet: Removed LNI 192.168.204.125@tcp135 [ 6824.978612] Key type .llcrypt unregistered [ 6824.980578] Key type ._llcrypt unregistered [ 6834.724295] Key type ._llcrypt registered [ 6834.726300] Key type .llcrypt registered [ 6834.794071] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 6842.974719] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 6843.263619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6843.297238] alg: No test for adler32 (adler32-zlib) [ 6844.170390] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 6844.250067] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 6845.855381] Key type lgssc registered [ 6846.495203] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6863.718652] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 21:44:37 (1777599877) [ 6871.931540] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 6873.409839] Key type lgssc unregistered [ 6873.590664] LNet: 441915:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6873.595586] LNetError: 441915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6873.608540] LNet: Removed LNI 192.168.204.125@tcp [ 6874.006704] Key type .llcrypt unregistered [ 6874.008358] Key type ._llcrypt unregistered [ 6882.593830] Key type ._llcrypt registered [ 6882.596493] Key type .llcrypt registered [ 6882.662420] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 6883.005273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6883.049256] alg: No test for adler32 (adler32-zlib) [ 6883.943430] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 6884.065981] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 6885.671157] Key type lgssc registered [ 6886.280409] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6891.033251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6891.045380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6892.184922] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6892.204726] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6892.258787] Lustre: lustre-MDT0000: new disk, initializing [ 6892.292898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6892.302654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6894.306310] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6899.971706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6900.047537] Lustre: 443089:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6900.092898] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6900.096702] Lustre: Skipped 1 previous similar message [ 6900.144215] Lustre: lustre-MDT0001: new disk, initializing [ 6900.191935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6900.213132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6900.219471] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6902.072197] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6906.190676] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6907.954820] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6911.187574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6911.368099] Lustre: lustre-OST0000: new disk, initializing [ 6911.371577] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6911.429813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6911.498550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6911.501485] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6911.548258] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6914.148278] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6918.141516] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6919.760259] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing lnet_if_list [ 6921.519426] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 6921.531874] LNet: Added LNI 192.168.204.125@tcp10 [8/256/0/180] [ 6923.239704] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 6923.248314] LNet: Added LNI 192.168.204.125@tcp11 [8/256/0/180] [ 6924.420715] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 6924.435351] LNet: Added LNI 192.168.204.125@tcp12 [8/256/0/180] [ 6925.771458] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 6925.785106] LNet: Added LNI 192.168.204.125@tcp13 [8/256/0/180] [ 6927.048162] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 6927.057164] LNet: Added LNI 192.168.204.125@tcp14 [8/256/0/180] [ 6928.365446] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 6928.375424] LNet: Added LNI 192.168.204.125@tcp15 [8/256/0/180] [ 6929.697162] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 6929.706362] LNet: Added LNI 192.168.204.125@tcp16 [8/256/0/180] [ 6931.031413] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 6931.043560] LNet: Added LNI 192.168.204.125@tcp17 [8/256/0/180] [ 6932.503773] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 6932.514608] LNet: Added LNI 192.168.204.125@tcp18 [8/256/0/180] [ 6934.161261] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 6934.170387] LNet: Added LNI 192.168.204.125@tcp19 [8/256/0/180] [ 6935.690840] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 6935.699649] LNet: Added LNI 192.168.204.125@tcp20 [8/256/0/180] [ 6937.251848] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 6937.263180] LNet: Added LNI 192.168.204.125@tcp21 [8/256/0/180] [ 6939.018622] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 6939.032442] LNet: Added LNI 192.168.204.125@tcp22 [8/256/0/180] [ 6940.673145] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 6940.687735] LNet: Added LNI 192.168.204.125@tcp23 [8/256/0/180] [ 6942.881508] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 6942.899321] LNet: Added LNI 192.168.204.125@tcp24 [8/256/0/180] [ 6945.246779] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 6945.264558] LNet: Added LNI 192.168.204.125@tcp25 [8/256/0/180] [ 6947.320017] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 6947.329384] LNet: Added LNI 192.168.204.125@tcp26 [8/256/0/180] [ 6949.760928] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 6949.779961] LNet: Added LNI 192.168.204.125@tcp27 [8/256/0/180] [ 6951.933223] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 6951.942403] LNet: Added LNI 192.168.204.125@tcp28 [8/256/0/180] [ 6953.540447] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 6953.556091] LNet: Added LNI 192.168.204.125@tcp29 [8/256/0/180] [ 6955.930261] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 6955.949195] LNet: Added LNI 192.168.204.125@tcp30 [8/256/0/180] [ 6958.423949] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 6958.442077] LNet: Added LNI 192.168.204.125@tcp31 [8/256/0/180] [ 6960.757072] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 6960.775925] LNet: Added LNI 192.168.204.125@tcp32 [8/256/0/180] [ 6962.692470] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 6962.701192] LNet: Added LNI 192.168.204.125@tcp33 [8/256/0/180] [ 6964.657359] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 6964.669674] LNet: Added LNI 192.168.204.125@tcp34 [8/256/0/180] [ 6967.120089] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 6967.129205] LNet: Added LNI 192.168.204.125@tcp35 [8/256/0/180] [ 6969.065329] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 6969.077051] LNet: Added LNI 192.168.204.125@tcp36 [8/256/0/180] [ 6971.489333] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 6971.507914] LNet: Added LNI 192.168.204.125@tcp37 [8/256/0/180] [ 6973.452333] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 6973.461373] LNet: Added LNI 192.168.204.125@tcp38 [8/256/0/180] [ 6975.770270] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 6975.782487] LNet: Added LNI 192.168.204.125@tcp39 [8/256/0/180] [ 6977.944791] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 6977.953772] LNet: Added LNI 192.168.204.125@tcp40 [8/256/0/180] [ 6980.321960] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 6980.339104] LNet: Added LNI 192.168.204.125@tcp41 [8/256/0/180] [ 6982.818956] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 6982.828446] LNet: Added LNI 192.168.204.125@tcp42 [8/256/0/180] [ 6985.679172] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 6985.699927] LNet: Added LNI 192.168.204.125@tcp43 [8/256/0/180] [ 6989.089145] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 6989.106248] LNet: Added LNI 192.168.204.125@tcp44 [8/256/0/180] [ 6992.139393] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 6992.155820] LNet: Added LNI 192.168.204.125@tcp45 [8/256/0/180] [ 6995.687523] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 6995.707546] LNet: Added LNI 192.168.204.125@tcp46 [8/256/0/180] [ 6999.167327] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 6999.185076] LNet: Added LNI 192.168.204.125@tcp47 [8/256/0/180] [ 7002.945155] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7002.965156] LNet: Added LNI 192.168.204.125@tcp48 [8/256/0/180] [ 7006.160245] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7006.175174] LNet: Added LNI 192.168.204.125@tcp49 [8/256/0/180] [ 7009.691424] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7009.710842] LNet: Added LNI 192.168.204.125@tcp50 [8/256/0/180] [ 7023.072334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7023.074818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7023.079252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7024.099888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7024.100584] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7024.102940] Lustre: Skipped 1 previous similar message [ 7024.108831] Lustre: Skipped 1 previous similar message [ 7025.583673] Lustre: server umount lustre-MDT0000 complete [ 7026.869476] LustreError: 443082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777600040 with bad export cookie 4916963373718062131 [ 7026.874651] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7026.875356] LustreError: 443082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7027.000514] Lustre: server umount lustre-MDT0001 complete [ 7033.873785] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 7036.441953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7045.599126] Lustre: 442362:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777600043/real 1777600043] req@000000009918fcde x1863948590733440/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777600059 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7045.606240] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7047.647109] Lustre: 442362:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777600045/real 1777600045] req@00000000c68b35a7 x1863948590733568/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777600061 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7050.719094] Lustre: 442361:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777600048/real 1777600048] req@00000000ffce8beb x1863948590733696/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777600064 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7051.743295] LustreError: 442359:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000047963112 x1863948590734592/t0(0) o250->MGC192.168.204.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7051.858198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7052.895127] Lustre: 442361:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777600050/real 1777600050] req@00000000cd5442d7 x1863948590733952/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777600066 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7052.940509] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7055.384933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7056.638327] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7056.959099] Lustre: 442362:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777600055/real 1777600055] req@000000007e681659 x1863948590734080/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1777600071 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7058.151959] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7059.177075] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7060.964087] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7064.036748] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7066.080755] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7066.083852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7066.090033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7066.092661] Lustre: Skipped 1 previous similar message [ 7067.104300] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7069.154475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7069.156291] Lustre: Skipped 2 previous similar messages [ 7071.892215] Lustre: server umount lustre-OST0000 complete [ 7074.272667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7074.273299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7074.276431] Lustre: Skipped 2 previous similar messages [ 7079.392213] LustreError: 468183:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7079.398821] LustreError: 468183:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 7079.400418] Lustre: server umount lustre-MDT0000 complete [ 7080.646222] LustreError: 443081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777600094 with bad export cookie 4916963373718064217 [ 7080.648975] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7080.650338] LustreError: 443081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7080.779602] Lustre: server umount lustre-MDT0001 complete [ 7084.063108] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 7085.032503] Key type lgssc unregistered [ 7085.188410] LNet: 470712:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7085.191242] LNetError: 470712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7085.193241] LNet: Removed LNI 192.168.204.125@tcp [ 7085.194297] LNet: Removed LNI 192.168.204.125@tcp10 [ 7085.195339] LNet: Removed LNI 192.168.204.125@tcp11 [ 7085.196451] LNet: Removed LNI 192.168.204.125@tcp12 [ 7085.197554] LNet: Removed LNI 192.168.204.125@tcp13 [ 7085.198712] LNet: Removed LNI 192.168.204.125@tcp14 [ 7085.199772] LNet: Removed LNI 192.168.204.125@tcp15 [ 7085.200841] LNet: Removed LNI 192.168.204.125@tcp16 [ 7085.202114] LNet: Removed LNI 192.168.204.125@tcp17 [ 7085.203346] LNet: Removed LNI 192.168.204.125@tcp18 [ 7085.204564] LNet: Removed LNI 192.168.204.125@tcp19 [ 7085.205750] LNet: Removed LNI 192.168.204.125@tcp20 [ 7085.206992] LNet: Removed LNI 192.168.204.125@tcp21 [ 7085.208291] LNet: Removed LNI 192.168.204.125@tcp22 [ 7085.209581] LNet: Removed LNI 192.168.204.125@tcp23 [ 7085.210877] LNet: Removed LNI 192.168.204.125@tcp24 [ 7085.211927] LNet: Removed LNI 192.168.204.125@tcp25 [ 7085.213211] LNet: Removed LNI 192.168.204.125@tcp26 [ 7085.214583] LNet: Removed LNI 192.168.204.125@tcp27 [ 7085.215831] LNet: Removed LNI 192.168.204.125@tcp28 [ 7085.217049] LNet: Removed LNI 192.168.204.125@tcp29 [ 7085.218300] LNet: Removed LNI 192.168.204.125@tcp30 [ 7085.219559] LNet: Removed LNI 192.168.204.125@tcp31 [ 7085.220709] LNet: Removed LNI 192.168.204.125@tcp32 [ 7085.221862] LNet: Removed LNI 192.168.204.125@tcp33 [ 7085.223022] LNet: Removed LNI 192.168.204.125@tcp34 [ 7085.224196] LNet: Removed LNI 192.168.204.125@tcp35 [ 7085.225451] LNet: Removed LNI 192.168.204.125@tcp36 [ 7085.226487] LNet: Removed LNI 192.168.204.125@tcp37 [ 7085.227604] LNet: Removed LNI 192.168.204.125@tcp38 [ 7085.228730] LNet: Removed LNI 192.168.204.125@tcp39 [ 7085.229814] LNet: Removed LNI 192.168.204.125@tcp40 [ 7085.230919] LNet: Removed LNI 192.168.204.125@tcp41 [ 7085.232039] LNet: Removed LNI 192.168.204.125@tcp42 [ 7085.233160] LNet: Removed LNI 192.168.204.125@tcp43 [ 7085.234298] LNet: Removed LNI 192.168.204.125@tcp44 [ 7085.235510] LNet: Removed LNI 192.168.204.125@tcp45 [ 7085.236686] LNet: Removed LNI 192.168.204.125@tcp46 [ 7085.237760] LNet: Removed LNI 192.168.204.125@tcp47 [ 7085.238905] LNet: Removed LNI 192.168.204.125@tcp48 [ 7085.240060] LNet: Removed LNI 192.168.204.125@tcp49 [ 7085.245650] LNet: Removed LNI 192.168.204.125@tcp50 [ 7085.517829] Key type .llcrypt unregistered [ 7085.518945] Key type ._llcrypt unregistered [ 7092.119164] Key type ._llcrypt registered [ 7092.120273] Key type .llcrypt registered [ 7092.157640] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 7097.958375] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 7098.250541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7098.288510] alg: No test for adler32 (adler32-zlib) [ 7099.140260] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 7099.218211] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 7100.799160] Key type lgssc registered [ 7101.170943] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7113.189207] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7113.198155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7114.303355] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7114.317053] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7114.356229] Lustre: lustre-MDT0000: new disk, initializing [ 7114.381099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7114.386814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7115.913496] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7121.096614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7121.155927] Lustre: 473845:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7121.193225] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7121.196442] Lustre: Skipped 1 previous similar message [ 7121.237797] Lustre: lustre-MDT0001: new disk, initializing [ 7121.280361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7121.298906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7121.305325] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7122.896220] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7126.460427] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7127.983637] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7130.196439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7130.284884] Lustre: *** cfs_fail_loc=90f, val=0*** [ 7130.313474] Lustre: lustre-OST0000: new disk, initializing [ 7130.315714] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7130.343992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7131.970983] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7134.758911] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7136.715666] Lustre: server umount lustre-OST0000 complete [ 7139.836406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7139.837772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7139.844663] Lustre: Skipped 1 previous similar message [ 7144.928900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7144.933137] Lustre: Skipped 2 previous similar messages [ 7149.024871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7149.028603] Lustre: Skipped 1 previous similar message [ 7152.095317] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7152.179159] Lustre: server umount lustre-MDT0000 complete [ 7153.896994] LustreError: 473839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777600168 with bad export cookie 12741839125988734543 [ 7153.903747] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7153.904882] LustreError: 473839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7154.055397] Lustre: server umount lustre-MDT0001 complete [ 7158.310828] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 7159.755586] Key type lgssc unregistered [ 7159.912693] LNet: 477404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7159.917789] LNetError: 477404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7159.928626] LNet: Removed LNI 192.168.204.125@tcp [ 7160.370976] Key type .llcrypt unregistered [ 7160.373178] Key type ._llcrypt unregistered [ 7169.060220] Key type ._llcrypt registered [ 7169.062098] Key type .llcrypt registered [ 7169.104831] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 7169.434183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7169.481926] alg: No test for adler32 (adler32-zlib) [ 7170.369745] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 7170.466096] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 7172.055181] Key type lgssc registered [ 7172.621284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7176.312769] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7176.320876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7177.484860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7178.973169] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7182.518094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7182.657106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7184.412552] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7186.939323] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7188.247304] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7191.402669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7191.537736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7192.550508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7192.553224] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7193.503333] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7196.082951] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7202.798738] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7212.512972] loop0: detected capacity change from 0 to 204800000 [ 7212.559763] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7212.562976] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7212.566181] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7215.014388] loop0: detected capacity change from 0 to 204800000 [ 7215.103920] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7215.108732] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7215.115203] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7217.554918] loop0: detected capacity change from 0 to 204800000 [ 7219.747175] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7219.812452] Lustre: fs15246-OST0000: new disk, initializing [ 7219.815339] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7219.841068] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7222.451122] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7227.332338] loop1: detected capacity change from 0 to 204800000 [ 7229.565123] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7229.626697] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7229.664154] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7229.714180] Lustre: fs15246-MDT0000: new disk, initializing [ 7229.790366] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7229.806603] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7229.871485] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7230.879912] Lustre: 483016:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000006f6c4b3a x1863948890572416/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1777600260 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7231.472256] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7238.947925] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7238.954274] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7238.970296] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7245.286563] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7245.296043] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7247.083291] Lustre: server umount fs15246-MDT0000 complete [ 7250.405269] LustreError: 480407:0:(ldlm_lib.c:1180:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7250.414540] LustreError: 480407:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 7250.990730] Lustre: server umount fs15246-OST0000 complete [ 7255.524083] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7255.524910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7255.531330] Lustre: Skipped 1 previous similar message [ 7255.537681] Lustre: Skipped 1 previous similar message [ 7260.641317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7260.645246] Lustre: Skipped 1 previous similar message [ 7261.543508] Lustre: server umount lustre-OST0000 complete [ 7264.736414] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7264.741460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7264.750068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7269.477127] Lustre: server umount lustre-MDT0000 complete [ 7270.880198] LustreError: 478633:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7270.888032] LustreError: 478633:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 7271.215509] LustreError: 478614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777600285 with bad export cookie 3234851970184403907 [ 7271.220070] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7271.222989] LustreError: 478614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7271.372152] Lustre: server umount lustre-MDT0001 complete [ 7275.660092] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 7276.958389] Key type lgssc unregistered [ 7277.131516] LNet: 486072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7277.136655] LNetError: 486072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7277.149578] LNet: Removed LNI 192.168.204.125@tcp [ 7277.592586] Key type .llcrypt unregistered [ 7277.595041] Key type ._llcrypt unregistered [ 7287.205702] Key type ._llcrypt registered [ 7287.207946] Key type .llcrypt registered [ 7287.264974] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 7287.568026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7287.620045] alg: No test for adler32 (adler32-zlib) [ 7288.486056] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 7288.575811] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 7290.159182] Key type lgssc registered [ 7290.722528] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7306.749397] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 7311.393959] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 7327.750663] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7327.766070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7328.877601] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7328.895384] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7328.943736] Lustre: lustre-MDT0000: new disk, initializing [ 7328.977464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7328.987921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7330.743467] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7336.164219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7336.230944] Lustre: 490455:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7336.270879] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7336.274907] Lustre: Skipped 1 previous similar message [ 7336.322709] Lustre: lustre-MDT0001: new disk, initializing [ 7336.363870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7336.382543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7336.389057] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7338.113593] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7342.103502] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7343.857845] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7347.047036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7347.208543] Lustre: lustre-OST0000: new disk, initializing [ 7347.212273] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7347.259990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7349.640836] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7353.240454] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7353.350322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7353.355810] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7353.393234] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7355.036814] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7355.153434] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7356.731038] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7356.814902] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7358.434528] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7358.436884] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7358.439036] LustreError: Skipped 1 previous similar message [ 7358.447226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7358.452713] Lustre: Skipped 1 previous similar message [ 7359.785382] Lustre: server umount lustre-OST0000 complete [ 7362.016428] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7362.021617] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7362.029612] Lustre: Skipped 1 previous similar message [ 7362.033281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7363.553162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7363.554565] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7367.732612] Lustre: server umount lustre-MDT0000 complete [ 7369.532877] LustreError: 493592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777600383 with bad export cookie 4958234773710003960 [ 7369.538465] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7369.540548] LustreError: 493592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7369.693512] Lustre: server umount lustre-MDT0001 complete [ 7378.365573] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 7381.455338] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 7390.870127] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7233 sec ========== 21:53:24 (1777600404) [ 7391.366983] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:53:25 (1777600405) === [ 7392.547939] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:53:26 (1777600406) === [ 7408.247497] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 7409.285756] Key type lgssc unregistered [ 7409.412295] LNet: 498633:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7409.416034] LNetError: 498633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7409.425428] LNet: Removed LNI 192.168.204.125@tcp [ 7409.718887] Key type .llcrypt unregistered [ 7409.720523] Key type ._llcrypt unregistered