[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 561442231 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002365] x2apic enabled [ 0.003020] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008304] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.010027] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.011014] pid_max: default: 32768 minimum: 301 [ 0.012152] LSM: Security Framework initializing [ 0.013068] Yama: becoming mindful. [ 0.015006] SELinux: Initializing. [ 0.016099] *** VALIDATE selinux *** [ 0.024509] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029317] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030181] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031118] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032123] *** VALIDATE tmpfs *** [ 0.033467] *** VALIDATE proc *** [ 0.035103] *** VALIDATE cgroup *** [ 0.036011] *** VALIDATE cgroup2 *** [ 0.038041] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039153] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041026] Spectre V2 : User space: Vulnerable [ 0.042009] Speculative Store Bypass: Vulnerable [ 0.045540] debug: unmapping init [mem 0xffffffff9a659000-0xffffffff9a660fff] [ 0.047235] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048721] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049023] ... version: 2 [ 0.050016] ... bit width: 48 [ 0.051016] ... generic registers: 4 [ 0.052013] ... value mask: 0000ffffffffffff [ 0.053018] ... max period: 00007fffffffffff [ 0.054017] ... fixed-purpose events: 3 [ 0.055015] ... event mask: 000000070000000f [ 0.057226] rcu: Hierarchical SRCU implementation. [ 0.059528] smp: Bringing up secondary CPUs ... [ 0.060644] x86: Booting SMP configuration: [ 0.061032] .... node #0, CPUs: #1 #2 #3 [ 0.065012] smp: Brought up 1 node, 4 CPUs [ 0.067022] smpboot: Max logical packages: 1 [ 0.068015] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.156408] node 0 deferred pages initialised in 85ms [ 0.161087] devtmpfs: initialized [ 0.163270] x86/mm: Memory block size: 128MB [ 0.167071] gcov: version magic: 0x41383552 [ 0.170357] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.171139] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.172332] pinctrl core: initialized pinctrl subsystem [ 0.173221] [ 0.173788] ************************************************************* [ 0.174014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.175014] ** ** [ 0.176014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.177015] ** ** [ 0.178012] ** This means that this kernel is built to expose internal ** [ 0.179014] ** IOMMU data structures, which may compromise security on ** [ 0.180013] ** your system. ** [ 0.181011] ** ** [ 0.182013] ** If you see this message and you are not debugging the ** [ 0.183017] ** kernel, report this immediately to your vendor! ** [ 0.184015] ** ** [ 0.185013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.186017] ************************************************************* [ 0.187389] NET: Registered protocol family 16 [ 0.189358] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.191078] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.194062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.200010] cpuidle: using governor menu [ 0.202695] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.205533] PCI: Using configuration type 1 for base access [ 0.207139] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.217056] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.219052] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.225034] cryptd: max_cpu_qlen set to 1000 [ 0.228264] ACPI: Added _OSI(Module Device) [ 0.230014] ACPI: Added _OSI(Processor Device) [ 0.232015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.234016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.239185] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.246438] ACPI: Interpreter enabled [ 0.248069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.249011] ACPI: Using IOAPIC for interrupt routing [ 0.251168] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.255449] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.265987] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.268049] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.271022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.274098] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.280301] acpiphp: Slot [2] registered [ 0.281151] acpiphp: Slot [5] registered [ 0.282162] acpiphp: Slot [6] registered [ 0.284109] acpiphp: Slot [7] registered [ 0.285151] acpiphp: Slot [8] registered [ 0.287134] acpiphp: Slot [9] registered [ 0.288114] acpiphp: Slot [10] registered [ 0.290125] acpiphp: Slot [3] registered [ 0.291097] acpiphp: Slot [4] registered [ 0.293119] acpiphp: Slot [11] registered [ 0.294164] acpiphp: Slot [12] registered [ 0.296126] acpiphp: Slot [13] registered [ 0.298119] acpiphp: Slot [14] registered [ 0.299177] acpiphp: Slot [15] registered [ 0.301115] acpiphp: Slot [16] registered [ 0.302166] acpiphp: Slot [17] registered [ 0.304137] acpiphp: Slot [18] registered [ 0.306121] acpiphp: Slot [19] registered [ 0.307122] acpiphp: Slot [20] registered [ 0.309118] acpiphp: Slot [21] registered [ 0.311078] acpiphp: Slot [22] registered [ 0.312116] acpiphp: Slot [23] registered [ 0.314110] acpiphp: Slot [24] registered [ 0.316107] acpiphp: Slot [25] registered [ 0.317131] acpiphp: Slot [26] registered [ 0.319184] acpiphp: Slot [27] registered [ 0.320087] acpiphp: Slot [28] registered [ 0.322117] acpiphp: Slot [29] registered [ 0.323182] acpiphp: Slot [30] registered [ 0.324146] acpiphp: Slot [31] registered [ 0.326071] PCI host bridge to bus 0000:00 [ 0.328020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.330023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.333034] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.336033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.338022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.342036] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.344194] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.348052] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.351416] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.361817] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.366043] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.369021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.371020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.374019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.377465] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.380177] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.382049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.386111] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.393019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.407828] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.412009] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.419711] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.429020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.437025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.461021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.479043] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.486029] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.496017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.521030] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.539015] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.549017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.556018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.581021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.597421] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.605031] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.614030] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.639035] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.653665] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.666019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.683032] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.718020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.732946] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.762043] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.771028] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.798032] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.807878] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.810280] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.811283] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.813413] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.815156] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.819189] iommu: Default domain type: Passthrough [ 0.820414] SCSI subsystem initialized [ 0.822077] ACPI: bus type USB registered [ 0.823000] usbcore: registered new interface driver usbfs [ 0.823000] usbcore: registered new interface driver hub [ 0.825103] usbcore: registered new device driver usb [ 0.826126] pps_core: LinuxPPS API ver. 1 registered [ 0.828016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.830095] PTP clock support registered [ 0.831206] EDAC MC: Ver: 3.0.0 [ 0.833144] PCI: Using ACPI for IRQ routing [ 0.834682] NetLabel: Initializing [ 0.836012] NetLabel: domain hash size = 128 [ 0.836917] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.837066] NetLabel: unlabeled traffic allowed by default [ 0.839140] vgaarb: loaded [ 0.840206] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.841007] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.846005] clocksource: Switched to clocksource kvm-clock [ 0.934824] VFS: Disk quotas dquot_6.6.0 [ 0.935877] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.937591] *** VALIDATE ramfs *** [ 0.938483] *** VALIDATE hugetlbfs *** [ 0.939574] pnp: PnP ACPI init [ 0.941203] pnp: PnP ACPI: found 6 devices [ 0.954260] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.956231] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.957727] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.958967] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.960366] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.961761] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.963443] NET: Registered protocol family 2 [ 0.965262] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.969411] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.971824] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.977135] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.980061] TCP: Hash tables configured (established 65536 bind 65536) [ 0.982827] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.986222] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.989313] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.992473] NET: Registered protocol family 1 [ 0.995399] RPC: Registered named UNIX socket transport module. [ 0.998106] RPC: Registered udp transport module. [ 1.000202] RPC: Registered tcp transport module. [ 1.002341] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.005224] NET: Registered protocol family 44 [ 1.006809] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.009195] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.011487] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.014535] PCI: CLS 0 bytes, default 64 [ 1.016508] Unpacking initramfs... [ 2.456962] debug: unmapping init [mem 0xffff9e2b7cc54000-0xffff9e2b7ffbffff] [ 2.459995] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.462343] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.465565] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.988576] Initialise system trusted keyrings [ 2.989742] Key type blacklist registered [ 2.991590] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.002513] zbud: loaded [ 3.007166] *** VALIDATE nfs *** [ 3.008509] *** VALIDATE nfs4 *** [ 3.009766] pstore: using deflate compression [ 3.013313] Platform Keyring initialized [ 3.123481] NET: Registered protocol family 38 [ 3.126085] Key type asymmetric registered [ 3.127902] Asymmetric key parser 'x509' registered [ 3.130125] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.133579] io scheduler mq-deadline registered [ 3.135581] io scheduler kyber registered [ 3.137533] io scheduler bfq registered [ 3.139466] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.143178] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.146187] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.149396] ACPI: Power Button [PWRF] [ 3.155217] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.162334] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.180326] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.191076] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.207313] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.233708] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.262389] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.267654] Non-volatile memory driver v1.3 [ 3.269322] Linux agpgart interface v0.103 [ 3.298609] virtio_blk virtio1: [vda] 134720 512-byte logical blocks (69.0 MB/65.8 MiB) [ 3.300778] vda: detected capacity change from 0 to 68976640 [ 3.317429] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.319913] vdb: detected capacity change from 0 to 1073741824 [ 3.334565] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.337780] vdc: detected capacity change from 0 to 2621440000 [ 3.352298] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.354649] vdd: detected capacity change from 0 to 2621440000 [ 3.373337] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.376642] vde: detected capacity change from 0 to 4294967296 [ 3.406663] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.408659] vdf: detected capacity change from 0 to 4294967296 [ 3.414531] libphy: Fixed MDIO Bus: probed [ 3.419560] usbcore: registered new interface driver usbserial_generic [ 3.421424] usbserial: USB Serial support registered for generic [ 3.423289] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.426764] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.428130] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.430239] mousedev: PS/2 mouse device common for all mice [ 3.432266] rtc_cmos 00:05: RTC can wake from S4 [ 3.434574] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.437430] rtc_cmos 00:05: registered as rtc0 [ 3.439965] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.441306] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.443642] intel_pstate: CPU model not supported [ 3.449405] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.453286] hid: raw HID events driver (C) Jiri Kosina [ 3.455879] usbcore: registered new interface driver usbhid [ 3.458523] usbhid: USB HID core driver [ 3.460565] drop_monitor: Initializing network drop monitor service [ 3.463565] Initializing XFRM netlink socket [ 3.465876] NET: Registered protocol family 10 [ 3.469184] Segment Routing with IPv6 [ 3.470384] NET: Registered protocol family 17 [ 3.472096] mpls_gso: MPLS GSO support [ 3.476623] RAS: Correctable Errors collector initialized. [ 3.478800] AVX version of gcm_enc/dec engaged. [ 3.480220] AES CTR mode by8 optimization enabled [ 3.554603] sched_clock: Marking stable (3554578761, 0)->(4547824096, -993245335) [ 3.558669] registered taskstats version 1 [ 3.560916] Loading compiled-in X.509 certificates [ 3.562935] zswap: loaded using pool lzo/zbud [ 3.588451] Key type big_key registered [ 3.601463] Key type encrypted registered [ 3.602657] ima: No TPM chip found, activating TPM-bypass! [ 3.608377] ima: Allocated hash algorithm: sha1 [ 3.610250] ima: No architecture policies found [ 3.616442] evm: Initialising EVM extended attributes: [ 3.618049] evm: security.selinux [ 3.618836] evm: security.ima [ 3.619570] evm: security.capability [ 3.620373] evm: HMAC attrs: 0x1 [ 3.622316] rtc_cmos 00:05: setting system clock to 2026-03-02 17:40:57 UTC (1772473257) [ 3.630852] debug: unmapping init [mem 0xffffffff9b603000-0xffffffff9b7fffff] [ 3.632954] debug: unmapping init [mem 0xffffffff9a382000-0xffffffff9a658fff] [ 3.639151] Write protecting the kernel read-only data: 28672k [ 3.641925] debug: unmapping init [mem 0xffffffff98a03000-0xffffffff98bfffff] [ 3.644504] debug: unmapping init [mem 0xffffffff99314000-0xffffffff993fffff] [ 3.693223] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.699529] systemd[1]: Detected virtualization kvm. [ 3.701509] systemd[1]: Detected architecture x86-64. [ 3.703429] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.728852] systemd[1]: No hostname configured. [ 3.732300] systemd[1]: Set hostname to . [ 3.734844] random: systemd: uninitialized urandom read (16 bytes read) [ 3.736483] systemd[1]: Initializing machine ID from random generator. [ 3.784287] random: ln: uninitialized urandom read (6 bytes read) [ 3.881501] random: systemd: uninitialized urandom read (16 bytes read) [ 3.884103] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.888647] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.893419] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. Starting Setup Virtual Console... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.536137] device-mapper: uevent: version 1.0.3 [ 4.538820] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.185828] random: fast init done [ 5.192151] virtio_net virtio0 ens2: renamed from eth0 [ 5.255682] scsi host0: ata_piix [ 5.272365] scsi host1: ata_piix [ 5.274088] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.276388] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.982917] dracut-initqueue[583]: RTNETLINK answers: File exists [ 10.129290] random: crng init done [ 10.130537] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.524766] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.632940] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.877606] SELinux: Disabled at runtime. [ 11.940949] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.950279] systemd[1]: Detected virtualization kvm. [ 11.952313] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.418213] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.421267] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.426151] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.431306] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.433635] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.440380] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.458554] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [ 12.632848] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.927119] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.238458] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.296313] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.424095] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.432743] EDAC sbridge: Ver: 1.1.2 [ 14.763186] Key type dns_resolver registered [ 15.084879] NFS: Registering the id_resolver key type [ 15.086365] Key type id_resolver registered [ 15.087846] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg458-server login: [ 47.874335] libcfs: loading out-of-tree module taints kernel. [ 47.915772] Key type ._llcrypt registered [ 47.917714] Key type .llcrypt registered [ 48.016845] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 73.846113] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 75.478406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 75.505415] alg: No test for adler32 (adler32-zlib) [ 77.115960] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 77.939983] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 79.776571] Key type lgssc registered [ 82.160514] Lustre: Echo OBD driver; http://www.lustre.org/ [ 99.609270] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 143.472915] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 156.571873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 156.623516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 158.030345] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 158.093545] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 158.240105] Lustre: lustre-MDT0000: new disk, initializing [ 158.403450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 158.433449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 162.884339] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 177.525079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 177.655909] Lustre: 6501:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 177.718875] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 177.724824] Lustre: Skipped 1 previous similar message [ 177.902201] Lustre: lustre-MDT0001: new disk, initializing [ 177.998830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 178.043737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 178.063347] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 182.827883] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 188.219912] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 197.769721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 197.977224] Lustre: lustre-OST0000: new disk, initializing [ 197.983780] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 198.035359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 200.094084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 200.118356] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 200.178317] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 203.804324] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.088250] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 217.161709] Lustre: lustre-OST0001: new disk, initializing [ 217.164061] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 217.210601] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 222.389667] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.831184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 226.839549] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 226.915822] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 232.912216] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 241.582675] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 250.731926] hrtimer: interrupt took 4810099 ns [ 252.529926] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing check_logdir /tmp/testlogs/ [ 257.140978] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing yml_node [ 261.615906] Lustre: DEBUG MARKER: Client: 2.17.50.196 [ 264.243160] Lustre: DEBUG MARKER: MDS: 2.17.50.196 [ 266.722834] Lustre: DEBUG MARKER: OSS: 2.17.50.196 [ 268.163862] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Mar 2 12:45:19 EST 2026 [ 283.526574] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 284.817328] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 301.023990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 301.033251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 301.040282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 303.584519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 303.585374] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 303.586968] Lustre: Skipped 1 previous similar message [ 303.606742] Lustre: Skipped 1 previous similar message [ 308.704323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 308.709312] Lustre: Skipped 5 previous similar messages [ 313.826178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 313.835167] Lustre: Skipped 3 previous similar messages [ 314.335254] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 314.518800] Lustre: server umount lustre-MDT0000 complete [ 317.719315] LustreError: 6494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772473571 with bad export cookie 5340083339559278388 [ 317.725760] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 317.727412] LustreError: 6494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 318.950378] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 318.961871] Lustre: Skipped 1 previous similar message [ 318.965865] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 318.972245] Lustre: Skipped 1 previous similar message [ 328.096280] LustreError: 6510:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 328.096443] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 328.116723] LustreError: 6510:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 328.137451] Lustre: Skipped 3 previous similar messages [ 332.255105] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 332.555739] Lustre: server umount lustre-MDT0001 complete [ 335.858328] Lustre: server umount lustre-OST0000 complete [ 338.563486] Lustre: server umount lustre-OST0001 complete [ 343.320257] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 350.581532] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 381.004431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 381.284455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 381.312892] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 381.388473] Lustre: lustre-MDT0000: new disk, initializing [ 381.478126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 381.492312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 384.934395] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 393.668436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 393.784739] Lustre: 14975:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 393.842200] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 393.844698] Lustre: Skipped 1 previous similar message [ 393.914823] Lustre: lustre-MDT0001: new disk, initializing [ 394.010631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 394.059056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 394.077448] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 397.400297] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 404.089619] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 407.214373] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 412.724145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 412.945841] Lustre: lustre-OST0000: new disk, initializing [ 412.949121] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 413.026439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 414.081266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 414.093695] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 414.210281] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 417.870897] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 424.233132] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 427.508167] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 427.654647] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 430.791430] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 430.907705] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 434.658753] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 434.663517] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 434.664339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 434.666063] Lustre: Skipped 2 previous similar messages [ 438.526106] Lustre: server umount lustre-OST0000 complete [ 444.899119] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 444.915063] Lustre: Skipped 1 previous similar message [ 447.595448] Lustre: server umount lustre-MDT0000 complete [ 450.021512] LustreError: 15762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 450.033544] LustreError: 15762:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 450.722441] LustreError: 14969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772473704 with bad export cookie 5340083339559279914 [ 450.733806] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 450.734975] LustreError: 14969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 451.067128] Lustre: server umount lustre-MDT0001 complete [ 454.488374] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 12:48:26 (1772473706) [ 456.377406] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 457.734511] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 12:48:29 (1772473709) [ 462.046562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 462.316621] LustreError: 18979:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 462.361961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 465.069465] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 467.428346] LustreError: 18980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 470.975522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 474.323809] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 478.152664] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 480.632228] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 485.162506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 489.098337] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 492.801224] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 501.729043] LustreError: lustre-OST0000-osc-MDT0001: operation ost_create to node 0@lo failed: rc = -107 [ 501.729445] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 501.735093] LustreError: Skipped 1 previous similar message [ 501.753118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 501.758604] Lustre: Skipped 3 previous similar messages [ 507.195821] Lustre: server umount lustre-OST0000 complete [ 516.297747] Lustre: server umount lustre-MDT0000 complete [ 517.088501] LustreError: 18980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 517.111746] LustreError: 18980:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 518.495189] LustreError: 18960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772473772 with bad export cookie 5340083339559280803 [ 518.496420] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 518.500126] LustreError: 18960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 518.514995] LustreError: 19690:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 518.518629] LustreError: 19690:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 518.713572] Lustre: server umount lustre-MDT0001 complete [ 526.075548] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 527.924416] Key type lgssc unregistered [ 528.101952] LNet: 22157:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 528.108339] LNetError: 22157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 528.128935] LNet: Removed LNI 192.168.204.158@tcp [ 528.717144] Key type .llcrypt unregistered [ 528.718810] Key type ._llcrypt unregistered [ 543.831274] Key type ._llcrypt registered [ 543.832858] Key type .llcrypt registered [ 543.905144] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 544.372752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 544.391266] alg: No test for adler32 (adler32-zlib) [ 545.387215] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 545.526822] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 547.128317] Key type lgssc registered [ 547.728973] Lustre: Echo OBD driver; http://www.lustre.org/ [ 552.915717] loop: module loaded [ 552.954225] loop0: detected capacity change from 0 to 204800000 [ 553.096508] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 553.105444] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 553.244218] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 556.486078] loop0: detected capacity change from 0 to 204800000 [ 556.623949] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 556.629451] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 556.640475] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 559.530859] loop0: detected capacity change from 0 to 204800000 [ 562.858404] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 562.872597] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 564.043613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 564.076139] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 564.138202] Lustre: lustre-MDT0000: new disk, initializing [ 564.178773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 564.187509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 566.757259] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 572.281789] loop1: detected capacity change from 0 to 204800000 [ 575.846029] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 576.067591] Lustre: lustre-OST07c6: new disk, initializing [ 576.070993] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 576.080235] Lustre: Skipped 1 previous similar message [ 576.135450] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 579.517903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 579.528306] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 579.557638] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 579.820755] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 583.039992] Lustre: 24288:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 583.047816] Lustre: 24288:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 583.050932] Lustre: 24288:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 583.056111] Lustre: 24288:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 583.059215] Lustre: 24288:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 583.061672] Lustre: 24288:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 597.530334] Lustre: 25781:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 597.533813] Lustre: 25781:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 597.539759] Lustre: 25781:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 597.552991] Lustre: 25781:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 597.563480] Lustre: 25781:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 597.568737] Lustre: 25781:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 601.836579] Lustre: 24288:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 601.845168] Lustre: 24288:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 1 previous similar message [ 601.855336] Lustre: 24288:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 601.863295] Lustre: 24288:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 601.874247] Lustre: 24288:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 601.878626] Lustre: 24288:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 601.883567] Lustre: 24288:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 601.890108] Lustre: 24288:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 601.898490] Lustre: 24288:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 601.902626] Lustre: 24288:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 601.910748] Lustre: 24288:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 601.918138] Lustre: 24288:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 605.099554] Lustre: 24286:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 605.107228] Lustre: 24286:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 1 previous similar message [ 605.110974] Lustre: 24286:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 605.114246] Lustre: 24286:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 605.118104] Lustre: 24286:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 605.123807] Lustre: 24286:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 605.128404] Lustre: 24286:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 605.133904] Lustre: 24286:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 605.138857] Lustre: 24286:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 605.144245] Lustre: 24286:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 605.148633] Lustre: 24286:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 605.153409] Lustre: 24286:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 609.411320] Lustre: 24287:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 609.415459] Lustre: 24287:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 1 previous similar message [ 609.418496] Lustre: 24287:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 609.422978] Lustre: 24287:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 609.427804] Lustre: 24287:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 609.431975] Lustre: 24287:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 609.435829] Lustre: 24287:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 609.447772] Lustre: 24287:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 609.453162] Lustre: 24287:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 609.457145] Lustre: 24287:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 609.461424] Lustre: 24287:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 609.466679] Lustre: 24287:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 619.556802] Lustre: 24286:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 619.561294] Lustre: 24286:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 5 previous similar messages [ 619.564865] Lustre: 24286:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 619.570100] Lustre: 24286:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 619.575121] Lustre: 24286:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 619.579892] Lustre: 24286:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 619.584261] Lustre: 24286:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 619.588215] Lustre: 24286:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 619.591521] Lustre: 24286:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 619.594931] Lustre: 24286:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 619.598580] Lustre: 24286:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 619.605124] Lustre: 24286:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 666.600789] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 666.610679] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 666.624130] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 666.790487] Lustre: server umount lustre-OST07c6 complete [ 672.443250] Lustre: server umount lustre-MDT0000 complete [ 683.473507] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 685.400671] Key type lgssc unregistered [ 685.629802] LNet: 27822:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 685.634523] LNetError: 27822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 685.649557] LNet: Removed LNI 192.168.204.158@tcp [ 686.187509] Key type .llcrypt unregistered [ 686.189840] Key type ._llcrypt unregistered [ 699.069873] Key type ._llcrypt registered [ 699.071723] Key type .llcrypt registered [ 699.146660] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 699.909035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 699.933961] alg: No test for adler32 (adler32-zlib) [ 700.907659] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 701.055771] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 702.671158] Key type lgssc registered [ 703.270869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 708.629195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 708.643470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 709.987291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 712.543720] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 719.026410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 719.337182] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 719.436367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 722.134200] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 725.567548] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 728.255933] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 732.949422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 733.177836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 734.187248] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 737.597351] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 741.522268] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 759.778315] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 759.779554] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 759.789882] LustreError: Skipped 1 previous similar message [ 759.791551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 759.798321] Lustre: Skipped 1 previous similar message [ 759.810499] Lustre: Skipped 1 previous similar message [ 761.260780] Lustre: server umount lustre-OST0000 complete [ 764.899635] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 764.903449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 764.912901] Lustre: Skipped 1 previous similar message [ 769.702393] Lustre: server umount lustre-MDT0000 complete [ 770.016371] LustreError: 30166:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 770.024960] LustreError: 30166:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 772.340048] LustreError: 29037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772474026 with bad export cookie 9744480854232355038 [ 772.345286] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 772.349681] LustreError: 29037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 772.587781] Lustre: server umount lustre-MDT0001 complete [ 779.788947] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 781.717397] Key type lgssc unregistered [ 781.981522] LNet: 32437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.987837] LNetError: 32437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 781.997677] LNet: Removed LNI 192.168.204.158@tcp [ 782.653523] Key type .llcrypt unregistered [ 782.654916] Key type ._llcrypt unregistered [ 795.422323] Key type ._llcrypt registered [ 795.428541] Key type .llcrypt registered [ 795.500442] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 796.107337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 796.159956] alg: No test for adler32 (adler32-zlib) [ 797.101360] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 797.237887] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 798.879195] Key type lgssc registered [ 799.585105] Lustre: Echo OBD driver; http://www.lustre.org/ [ 805.013808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 805.034208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 806.385528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 809.502181] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 815.271838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 815.449202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 818.548090] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 821.936765] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 824.455205] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 829.137334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 829.361453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 831.409890] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 833.228331] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 836.675990] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 850.917496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 850.923811] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 850.924757] Lustre: Skipped 1 previous similar message [ 850.925341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 853.564413] Lustre: server umount lustre-OST0000 complete [ 856.549279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 856.557310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 856.566553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 856.569782] Lustre: Skipped 1 previous similar message [ 861.153141] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 861.154764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 861.179505] Lustre: Skipped 1 previous similar message [ 862.552045] Lustre: server umount lustre-MDT0000 complete [ 865.078665] LustreError: 33648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772474118 with bad export cookie 1226858613082697715 [ 865.085500] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 865.089162] LustreError: 33648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 865.311806] Lustre: server umount lustre-MDT0001 complete [ 873.064328] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 874.872485] Key type lgssc unregistered [ 875.038480] LNet: 36845:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 875.044061] LNetError: 36845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 876.070241] LNet: Removed LNI 192.168.204.158@tcp [ 876.733169] Key type .llcrypt unregistered [ 876.734826] Key type ._llcrypt unregistered [ 892.031491] Key type ._llcrypt registered [ 892.032937] Key type .llcrypt registered [ 892.108178] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 892.951460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 892.987570] alg: No test for adler32 (adler32-zlib) [ 893.945310] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 894.073632] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 895.695159] Key type lgssc registered [ 896.385954] Lustre: Echo OBD driver; http://www.lustre.org/ [ 901.315911] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 901.332164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 902.689964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 905.145863] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 911.088613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 911.316822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 914.142391] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 917.999806] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 920.504151] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 925.489132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 925.823608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 927.862105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 929.685894] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 933.275336] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 942.048691] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 942.054983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 942.066401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 942.586623] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 942.598077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 942.602521] Lustre: Skipped 1 previous similar message [ 944.767635] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.58@tcp (stopping) [ 947.196361] Lustre: server umount lustre-MDT0000 complete [ 947.680703] LustreError: 38937:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 947.693121] LustreError: 38937:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 949.717087] LustreError: 38206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772474203 with bad export cookie 6467914046405421353 [ 949.718213] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 949.727241] LustreError: 38206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 949.865029] LustreError: 38223:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 949.890639] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.58@tcp (stopping) [ 952.804572] LustreError: 38223:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 952.805546] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 952.819567] Lustre: Skipped 1 previous similar message [ 954.848628] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 954.848635] LustreError: 38224:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 954.851342] Lustre: Skipped 1 previous similar message [ 959.968865] LustreError: 38223:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 959.984724] LustreError: 38223:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 964.063188] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 964.279148] Lustre: server umount lustre-MDT0001 complete [ 977.595312] Lustre: server umount lustre-OST0000 complete [ 987.696164] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 989.505495] Key type lgssc unregistered [ 989.675968] LNet: 41735:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 989.681197] LNetError: 41735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 989.689184] LNet: Removed LNI 192.168.204.158@tcp [ 990.183771] Key type .llcrypt unregistered [ 990.186250] Key type ._llcrypt unregistered [ 1004.863515] Key type ._llcrypt registered [ 1004.865592] Key type .llcrypt registered [ 1004.955765] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 1005.644710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1005.663613] alg: No test for adler32 (adler32-zlib) [ 1006.625699] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 1006.764364] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 1008.361405] Key type lgssc registered [ 1009.013828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1013.950188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1013.960489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1015.221803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1018.091800] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1023.135219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1023.457336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1026.281984] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1030.564700] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1033.158480] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1037.855136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1038.122159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1040.175353] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1042.967206] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1046.344441] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1053.665691] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1053.667738] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1053.679793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1053.687161] Lustre: Skipped 1 previous similar message [ 1057.806972] Lustre: server umount lustre-OST0000 complete [ 1074.149531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1074.151289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1074.161126] Lustre: Skipped 2 previous similar messages [ 1074.163330] Lustre: Skipped 1 previous similar message [ 1079.265975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1079.269943] Lustre: Skipped 2 previous similar messages [ 1079.405202] Lustre: server umount lustre-MDT0000 complete [ 1081.828067] LustreError: 43092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772474335 with bad export cookie 10374594213480527704 [ 1081.831791] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1081.835226] LustreError: 43092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1096.159431] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1096.311823] Lustre: server umount lustre-MDT0001 complete [ 1103.095745] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 1105.042387] Key type lgssc unregistered [ 1105.254977] LNet: 46464:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1105.266793] LNetError: 46464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1106.277632] LNet: Removed LNI 192.168.204.158@tcp [ 1106.850501] Key type .llcrypt unregistered [ 1106.851899] Key type ._llcrypt unregistered [ 1122.991912] Key type ._llcrypt registered [ 1122.993958] Key type .llcrypt registered [ 1123.098577] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 1123.836561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1123.888515] alg: No test for adler32 (adler32-zlib) [ 1124.858232] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 1125.043994] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 1126.663276] Key type lgssc registered [ 1127.461294] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1133.224396] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1133.259918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1134.649795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1137.431178] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1142.734749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1143.001816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1145.986572] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1149.242641] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1151.793484] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1156.125388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1156.367731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1159.480464] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1159.918035] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1163.729074] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1172.487978] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1174.423097] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1185.685667] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-03-0213h00m37s [ 1186.558835] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1188.321678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1188.326702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1188.330544] Lustre: Skipped 2 previous similar messages [ 1188.337447] Lustre: Skipped 2 previous similar messages [ 1189.344540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1189.347763] Lustre: Skipped 1 previous similar message [ 1192.042184] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.58@tcp (stopping) [ 1193.826176] Lustre: server umount lustre-MDT0000 complete [ 1195.490612] LustreError: 47843:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1195.499983] LustreError: 47843:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1195.591910] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1198.117166] LustreError: 47837:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1199.363030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1199.445726] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1199.574921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1199.617213] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1200.622874] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1201.688121] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1203.608764] Lustre: DEBUG MARKER: Wait for df (36040) ... [ 1204.476871] Lustre: DEBUG MARKER: done [ 1208.800226] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1208.802712] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1208.810376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1208.813886] Lustre: Skipped 1 previous similar message [ 1209.827284] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1209.831971] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1212.607977] Lustre: server umount lustre-OST0000 complete [ 1215.969953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1215.970470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1215.973756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1215.973767] Lustre: Skipped 3 previous similar messages [ 1215.999042] Lustre: Skipped 1 previous similar message [ 1227.233646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1227.240945] Lustre: Skipped 6 previous similar messages [ 1229.279620] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1229.399987] Lustre: server umount lustre-MDT0000 complete [ 1231.774964] LustreError: 50138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772474485 with bad export cookie 1427776249283238101 [ 1231.780941] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1231.784112] LustreError: 50138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1231.983465] Lustre: server umount lustre-MDT0001 complete [ 1238.468621] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 1240.193438] Key type lgssc unregistered [ 1240.371686] LNet: 52412:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1240.376609] LNetError: 52412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1240.385292] LNet: Removed LNI 192.168.204.158@tcp [ 1240.899964] Key type .llcrypt unregistered [ 1240.901989] Key type ._llcrypt unregistered [ 1262.764857] Key type ._llcrypt registered [ 1262.766693] Key type .llcrypt registered [ 1262.821927] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 1263.522437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1263.536177] alg: No test for adler32 (adler32-zlib) [ 1264.443281] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 1264.572427] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 1266.167183] Key type lgssc registered [ 1266.705949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1270.560216] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1270.575273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1271.740594] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1271.763140] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1271.847793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1274.105681] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1279.275533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1279.333519] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1279.349840] Lustre: 54581:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1279.497710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1281.967309] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1285.687432] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1287.651112] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1290.472225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1290.578713] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1290.681514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1293.619039] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1296.400477] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1297.449748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1301.451644] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-03-0213h02m33s [ 1302.281769] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1305.689731] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1306.828859] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1306.864969] LustreError: 56222:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1327.227979] Lustre: lustre-MDT0000: Client 09faf88c-190b-4f64-9238-e7390e9e6070 (at 192.168.204.58@tcp) reconnecting [ 1346.921069] LustreError: 56222:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1348.095213] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1349.364868] Lustre: DEBUG MARKER: done [ 1353.187854] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1353.190778] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1353.196873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1353.699676] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1353.702992] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1353.709689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1358.030346] Lustre: server umount lustre-OST0000 complete [ 1361.377921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1361.380407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1361.386787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1364.450922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1364.458532] Lustre: Skipped 4 previous similar messages [ 1366.292509] Lustre: server umount lustre-MDT0000 complete [ 1368.682903] LustreError: 54572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772474622 with bad export cookie 11500962601264221911 [ 1368.689159] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1368.692138] LustreError: 54572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1368.877368] Lustre: server umount lustre-MDT0001 complete [ 1374.523738] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 1375.992279] Key type lgssc unregistered [ 1376.169615] LNet: 58548:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1376.172390] LNetError: 58548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1376.181447] LNet: Removed LNI 192.168.204.158@tcp [ 1376.609990] Key type .llcrypt unregistered [ 1376.611597] Key type ._llcrypt unregistered [ 1397.945452] Key type ._llcrypt registered [ 1397.946775] Key type .llcrypt registered [ 1398.004560] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 1398.505544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1398.523709] alg: No test for adler32 (adler32-zlib) [ 1399.436623] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 1399.550603] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 1401.167188] Key type lgssc registered [ 1401.714364] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1403.344573] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1403.355795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1404.509194] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1404.525924] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1404.621994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1405.099468] Lustre: Failing over lustre-MDT0000 [ 1405.240651] Lustre: server umount lustre-MDT0000 complete [ 1408.513535] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 13:04:20 (1772474660) [ 1412.091679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1412.330506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1414.371864] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1419.045536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1419.116691] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1419.137521] Lustre: 61306:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1419.278772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1421.481906] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1424.684633] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1426.454144] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1429.382610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1429.475836] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1429.562219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1432.285479] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1435.061111] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1438.780183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1445.160746] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1448.931095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1448.931782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1448.939075] Lustre: Skipped 2 previous similar messages [ 1448.942552] Lustre: Skipped 1 previous similar message [ 1454.051926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1454.058200] Lustre: Skipped 3 previous similar messages [ 1454.309351] Lustre: server umount lustre-MDT0000 complete [ 1456.265416] LustreError: 61301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772474710 with bad export cookie 3948248702683395241 [ 1456.268246] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1456.272934] LustreError: 61301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1456.411643] Lustre: server umount lustre-MDT0001 complete [ 1457.992673] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1460.569808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1465.530014] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1466.344817] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x36caff46c0573d85 [ 1466.353568] Lustre: MGC192.168.204.158@tcp: Connection restored to 0@lo (at 0@lo) [ 1466.591953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1466.635488] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1468.596582] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1471.978081] LustreError: 62927:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1471.982118] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1471.992313] LustreError: 62927:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1473.259816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1475.408445] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1475.551435] Lustre: 60233:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772474713/real 1772474713] req@00000000e72039d7 x1858574381777408/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772474729 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1477.599140] Lustre: 60233:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772474715/real 1772474715] req@0000000078e88bba x1858574381777536/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772474731 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1477.851768] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1478.640507] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1479.656160] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1479.873445] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1480.671084] Lustre: 60234:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772474718/real 1772474718] req@000000009d68e6b9 x1858574381777664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772474734 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1483.295199] Lustre: 60235:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772474720/real 1772474720] req@000000005584a0da x1858574381777920/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772474736 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1483.745467] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1483.748396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1483.752868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1488.436662] Lustre: server umount lustre-MDT0000 complete [ 1489.894758] LustreError: 62926:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1489.900648] LustreError: 62926:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1490.052745] LustreError: 61299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772474743 with bad export cookie 3948248702683397509 [ 1490.055968] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1490.056906] LustreError: 61299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1490.216194] Lustre: server umount lustre-MDT0001 complete [ 1493.227835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1497.752624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1499.871214] LustreError: 67180:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 1499.876859] LustreError: 67180:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@000000009da22ae9 x1858574381798272/t0(0) o250->MGC192.168.204.158@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1772474753 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1499.899918] LustreError: 67180:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 1500.131302] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x36caff46c05742e8 [ 1500.137548] Lustre: MGC192.168.204.158@tcp: Connection restored to 0@lo (at 0@lo) [ 1500.305295] LustreError: 62927:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1500.346982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1500.349559] Lustre: Skipped 1 previous similar message [ 1500.379417] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1502.417674] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1505.763026] LustreError: 62926:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1505.763818] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1505.787192] Lustre: Skipped 2 previous similar messages [ 1506.733248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1508.918812] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1511.007258] Lustre: 60232:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772474748/real 1772474748] req@00000000be7091d2 x1858574381798528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772474764 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1511.844604] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1512.424083] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1513.766602] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1519.590516] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1519.591050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1519.601925] Lustre: Skipped 2 previous similar messages [ 1519.603709] Lustre: Skipped 4 previous similar messages [ 1523.532970] Lustre: server umount lustre-MDT0000 complete [ 1524.707210] LustreError: 67195:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1524.715600] LustreError: 67195:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1525.727231] LustreError: 61301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772474779 with bad export cookie 3948248702683398888 [ 1525.731943] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1525.738704] LustreError: 61301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1525.904882] Lustre: server umount lustre-MDT0001 complete [ 1529.810543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1531.917833] Lustre: DEBUG MARKER: files compared the same [ 1543.279543] Lustre: server umount lustre-OST0000 complete [ 1551.842608] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 1553.350802] Key type lgssc unregistered [ 1553.547167] LNet: 70226:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1553.565143] LNetError: 70226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1554.602862] LNet: Removed LNI 192.168.204.158@tcp [ 1555.140803] Key type .llcrypt unregistered [ 1555.143895] Key type ._llcrypt unregistered [ 1567.976753] Key type ._llcrypt registered [ 1567.978844] Key type .llcrypt registered [ 1568.050784] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 1568.672097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1568.691795] alg: No test for adler32 (adler32-zlib) [ 1569.607139] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 1569.768493] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 1571.399136] Key type lgssc registered [ 1571.990510] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1576.728809] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1576.747829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1578.024209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1580.641238] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug +malloc all [ 1586.036644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1586.195717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1588.502666] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug +malloc all [ 1591.709384] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1593.833796] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1597.491037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1597.645766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1600.504752] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug +malloc all [ 1602.798648] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1603.332213] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1617.380390] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1617.383337] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1617.387311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1619.064969] Lustre: server umount lustre-OST0000 complete [ 1621.983736] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1621.987639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1621.993790] Lustre: Skipped 1 previous similar message [ 1621.996749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1621.999795] Lustre: Skipped 1 previous similar message [ 1627.361301] Lustre: server umount lustre-MDT0000 complete [ 1627.616753] LustreError: 71606:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1627.622654] LustreError: 71606:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1629.346030] LustreError: 71586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772474883 with bad export cookie 13123815068253865897 [ 1629.352564] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1629.535960] Lustre: server umount lustre-MDT0001 complete [ 1634.979731] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 1636.484923] Key type lgssc unregistered [ 1636.675044] LNet: 74800:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1636.684487] LNetError: 74800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1636.700481] LNet: Removed LNI 192.168.204.158@tcp [ 1637.117902] Key type .llcrypt unregistered [ 1637.119224] Key type ._llcrypt unregistered [ 1650.480359] Key type ._llcrypt registered [ 1650.481507] Key type .llcrypt registered [ 1650.534372] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 1651.053961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1651.076330] alg: No test for adler32 (adler32-zlib) [ 1651.968091] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 1652.090767] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 1653.711226] Key type lgssc registered [ 1654.302169] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1658.347786] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1658.364098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1673.823448] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1678.943265] LustreError: 76152:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.158@tcp: failed processing log, type 4: rc = -110 [ 1705.632754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1708.224640] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1710.712456] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1714.676614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1714.770215] Lustre: *** cfs_fail_loc=706, val=0*** [ 1714.979860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1715.022383] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1717.178697] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1720.291343] LustreError: 76177:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1720.299923] LustreError: 76177:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1721.318526] LustreError: 76178:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1721.330247] LustreError: 76178:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1721.824994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1722.035475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1724.117705] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1726.851628] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1728.537357] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1730.528301] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1730.531437] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1730.535423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1732.576873] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1732.578743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1732.580710] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1735.583686] Lustre: server umount lustre-OST0000 complete [ 1737.696801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1737.697754] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1737.699563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1737.699569] Lustre: Skipped 1 previous similar message [ 1737.710394] Lustre: Skipped 1 previous similar message [ 1742.816903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1742.821168] Lustre: Skipped 2 previous similar messages [ 1743.742812] Lustre: server umount lustre-MDT0000 complete [ 1745.664838] LustreError: 76158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772474999 with bad export cookie 1476400073215186370 [ 1745.667100] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1745.674073] LustreError: 76158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1745.868955] Lustre: server umount lustre-MDT0001 complete [ 1751.154322] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 1752.591492] Key type lgssc unregistered [ 1752.771530] LNet: 79420:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1752.775787] LNetError: 79420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1752.794684] LNet: Removed LNI 192.168.204.158@tcp [ 1753.251391] Key type .llcrypt unregistered [ 1753.252895] Key type ._llcrypt unregistered [ 1767.265433] Key type ._llcrypt registered [ 1767.267055] Key type .llcrypt registered [ 1767.327867] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 1767.898066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1767.929823] alg: No test for adler32 (adler32-zlib) [ 1768.839495] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 1768.953704] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 1770.559495] Key type lgssc registered [ 1771.168728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1775.375124] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1775.387115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1778.133355] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1782.092266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1782.387815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1784.579043] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1787.872887] LustreError: 81465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1787.879505] LustreError: 81465:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1788.493153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1788.676501] LustreError: 81465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1788.693610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1791.251989] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1792.803603] LustreError: 81468:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1792.813779] LustreError: 81468:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1795.190655] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1795.197391] Lustre: MGS: Client fd427553-bd13-4433-b20b-3510bc15f5d6 (at 0@lo) reconnecting [ 1795.212560] LustreError: 81479:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1795.379637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1795.433337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1797.353936] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1810.917627] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1810.918111] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1810.931628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1810.941988] Lustre: Skipped 1 previous similar message [ 1813.493853] Lustre: server umount lustre-OST0000 complete [ 1816.032855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1816.034938] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1816.036825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1816.048485] Lustre: Skipped 2 previous similar messages [ 1816.053601] Lustre: Skipped 1 previous similar message [ 1821.153125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1821.159625] Lustre: Skipped 1 previous similar message [ 1821.532318] Lustre: server umount lustre-MDT0000 complete [ 1823.646534] Lustre: server umount lustre-MDT0001 complete [ 1825.501488] Lustre: server umount MGS complete [ 1831.137993] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 13:11:23 (1772475083) [ 1843.960444] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 1848.725269] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 1866.948597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1868.748587] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1873.387457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1873.501217] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1873.524529] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1873.567554] Lustre: lustre-MDT0001: new disk, initializing [ 1873.604329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1873.619519] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1873.622065] Lustre: 88528:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1873.626214] LustreError: 88528:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1875.621631] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1881.334276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1881.483991] Lustre: lustre-OST0000: new disk, initializing [ 1881.486345] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1881.523895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1884.594409] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1885.536053] Lustre: 89588:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000732b1505 x1858574768704384/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1772475155 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1888.705285] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1892.035547] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1892.045820] Lustre: MGS: Client eb863ba3-67f0-46b6-b21f-61407eebd02b (at 0@lo) reconnecting [ 1892.066222] Lustre: 87816:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1892.076555] LustreError: 88514:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1892.080608] LustreError: 88514:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1892.092215] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1892.139459] Lustre: lustre-MDT0000: new disk, initializing [ 1892.208681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1892.222255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1893.279586] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1893.283091] Lustre: 90238:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000000f00bc07 x1858574768710400/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1772475163 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1893.291256] Lustre: 90238:0:(client.c:1637:after_reply()) Skipped 1 previous similar message [ 1893.990843] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1902.373736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1902.378088] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1902.393434] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1926.623865] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1926.627687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1926.633483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1929.948957] Lustre: server umount lustre-OST0000 complete [ 1932.769131] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1932.772074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1932.776330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1932.776943] Lustre: Skipped 2 previous similar messages [ 1932.779057] Lustre: Skipped 2 previous similar messages [ 1937.996409] Lustre: server umount lustre-MDT0000 complete [ 1945.812317] Lustre: server umount lustre-MDT0001 complete [ 1949.430131] Lustre: server umount MGS complete [ 1954.421573] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 13:13:26 (1772475206) [ 1962.000223] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 1963.123130] Key type lgssc unregistered [ 1963.259379] LNet: 92698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1963.262344] LNetError: 92698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1963.269329] LNet: Removed LNI 192.168.204.158@tcp [ 1963.598463] Key type .llcrypt unregistered [ 1963.600040] Key type ._llcrypt unregistered [ 1971.799166] Key type ._llcrypt registered [ 1971.800521] Key type .llcrypt registered [ 1971.845377] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 1972.298620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1972.305086] alg: No test for adler32 (adler32-zlib) [ 1973.180197] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 1973.281354] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 1974.863207] Key type lgssc registered [ 1975.391124] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1978.773057] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1978.782860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1980.015549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1981.603212] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1985.230124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1985.386318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1986.990652] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1989.319647] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1990.827510] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1993.588749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1993.760797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1996.379269] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1998.663566] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2005.036505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2015.201855] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2015.203501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2015.205396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2015.209233] Lustre: Skipped 1 previous similar message [ 2018.005628] Lustre: server umount lustre-OST0000 complete [ 2020.321352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2020.322176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2020.325895] Lustre: Skipped 1 previous similar message [ 2020.327975] Lustre: Skipped 1 previous similar message [ 2025.440235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2025.442643] Lustre: Skipped 1 previous similar message [ 2025.568836] Lustre: server umount lustre-MDT0000 complete [ 2026.913329] LustreError: 96042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772475280 with bad export cookie 5684401767317879897 [ 2026.915950] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2026.917247] LustreError: 96042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2027.038617] Lustre: server umount lustre-MDT0001 complete [ 2031.130573] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 2032.358361] Key type lgssc unregistered [ 2032.489399] LNet: 97076:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2032.492276] LNetError: 97076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2032.505409] LNet: Removed LNI 192.168.204.158@tcp [ 2032.820794] Key type .llcrypt unregistered [ 2032.822730] Key type ._llcrypt unregistered [ 2039.709099] Key type ._llcrypt registered [ 2039.710499] Key type .llcrypt registered [ 2039.752713] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 2040.107680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2040.167611] alg: No test for adler32 (adler32-zlib) [ 2041.048516] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 2041.165107] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 2042.767216] Key type lgssc registered [ 2043.224121] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2045.694533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2045.698058] LustreError: 98120:0:(tgt_mount.c:2535:server_fill_super()) cfs_race id 716 sleeping [ 2045.700262] LustreError: 98121:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 waking [ 2045.703327] LustreError: 98120:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 2045.707078] LustreError: 98120:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2045.712702] LustreError: 98120:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2045.713681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2046.926051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2049.872711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2050.018107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2051.654049] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2053.643632] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing lsmod [ 2054.918218] LustreError: 99290:0:(tgt_mount.c:2535:server_fill_super()) cfs_race id 716 sleeping [ 2054.922518] LustreError: 99292:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 waking [ 2054.926447] LustreError: 99290:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 2054.929143] LustreError: 99290:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2054.933568] LustreError: 99290:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2054.940299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2055.112870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2056.174392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2059.491327] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2059.492943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2059.503141] Lustre: Skipped 2 previous similar messages [ 2059.507156] Lustre: Skipped 1 previous similar message [ 2062.558262] Lustre: server umount lustre-MDT0000 complete [ 2064.086185] LustreError: 98138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772475317 with bad export cookie 12435042568196650054 [ 2064.089129] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2064.090685] LustreError: 98138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2064.864923] LustreError: 98596:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2064.866192] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2064.871796] LustreError: 98596:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2064.885918] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2064.888446] Lustre: Skipped 1 previous similar message [ 2065.631495] LustreError: 98589:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c1bc984f x1858575054946816/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2065.698692] Lustre: server umount lustre-MDT0001 complete [ 2067.421099] Lustre: server umount lustre-OST0000 complete [ 2070.940879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2071.163651] LustreError: 100358:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2071.201247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2072.912278] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2076.626221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2076.640102] LustreError: 100359:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2076.781724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2078.539749] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2080.859940] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2082.323532] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2085.466375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2085.620681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2087.996871] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2090.435244] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2096.877623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 2107.361495] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2107.362488] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2107.364123] LustreError: Skipped 1 previous similar message [ 2107.364969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2107.374461] Lustre: Skipped 1 previous similar message [ 2109.688429] Lustre: server umount lustre-OST0000 complete [ 2112.480452] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2112.481325] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2112.484990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2112.489273] Lustre: Skipped 1 previous similar message [ 2112.494311] Lustre: Skipped 2 previous similar messages [ 2117.539485] Lustre: server umount lustre-MDT0000 complete [ 2117.600429] LustreError: 100358:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2117.608184] LustreError: 100358:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2119.356929] LustreError: 100339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772475373 with bad export cookie 12435042568196650943 [ 2119.358649] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2119.361373] LustreError: 100339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2119.499344] Lustre: server umount lustre-MDT0001 complete [ 2124.429117] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 2125.740186] Key type lgssc unregistered [ 2125.898720] LNet: 103554:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2125.901422] LNetError: 103554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2125.911331] LNet: Removed LNI 192.168.204.158@tcp [ 2126.289792] Key type .llcrypt unregistered [ 2126.291199] Key type ._llcrypt unregistered [ 2137.139896] Key type ._llcrypt registered [ 2137.141743] Key type .llcrypt registered [ 2137.195404] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 2137.657290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2137.694908] alg: No test for adler32 (adler32-zlib) [ 2138.620280] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 2138.755822] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 2140.359147] Key type lgssc registered [ 2140.945629] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2144.730806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2144.745790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2146.003033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2148.028867] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2152.057350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2152.249808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2153.986506] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2156.334726] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2157.875364] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2161.444990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2161.602867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2164.523802] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2167.005828] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2170.871398] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 2174.610507] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2175.963245] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2178.016294] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2178.019210] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2178.025922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2181.092699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2181.093780] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2181.100950] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2183.324113] Lustre: server umount lustre-OST0000 complete [ 2186.209096] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2186.215818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2186.218554] Lustre: Skipped 2 previous similar messages [ 2191.230631] Lustre: server umount lustre-MDT0000 complete [ 2191.327977] LustreError: 104933:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2191.333638] LustreError: 104933:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2192.650078] LustreError: 104913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772475446 with bad export cookie 12981132122763044880 [ 2192.652403] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2192.653925] LustreError: 104913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2192.773963] Lustre: server umount lustre-MDT0001 complete [ 2196.694054] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 2197.853867] Key type lgssc unregistered [ 2198.000448] LNet: 108211:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2198.004310] LNetError: 108211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2198.013832] LNet: Removed LNI 192.168.204.158@tcp [ 2198.359780] Key type .llcrypt unregistered [ 2198.361649] Key type ._llcrypt unregistered [ 2205.760511] Key type ._llcrypt registered [ 2205.761906] Key type .llcrypt registered [ 2205.807266] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 2206.277359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2206.291305] alg: No test for adler32 (adler32-zlib) [ 2207.179920] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 2207.272624] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 2208.863112] Key type lgssc registered [ 2209.301269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2212.535408] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2212.542276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2213.722405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2215.386566] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2219.344365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2219.565644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2221.404489] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2224.112518] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2225.917291] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2229.144261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2229.303625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2231.900668] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2234.196669] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2238.509835] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 2245.599891] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2245.603478] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2245.608388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2248.674112] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2248.674352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2248.674894] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2248.682934] Lustre: Skipped 1 previous similar message [ 2253.793631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2253.798610] Lustre: Skipped 1 previous similar message [ 2258.399353] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2258.475439] Lustre: server umount lustre-OST0000 complete [ 2260.447958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2260.452647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2260.460046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2264.034465] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2266.412923] Lustre: server umount lustre-MDT0000 complete [ 2268.138751] LustreError: 109373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772475522 with bad export cookie 9042875627961237081 [ 2268.143831] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2268.144520] LustreError: 109373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2268.293206] Lustre: server umount lustre-MDT0001 complete [ 2272.952892] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 2274.451542] Key type lgssc unregistered [ 2274.609738] LNet: 112572:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2274.615342] LNetError: 112572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2274.626975] LNet: Removed LNI 192.168.204.158@tcp [ 2275.044991] Key type .llcrypt unregistered [ 2275.047092] Key type ._llcrypt unregistered [ 2286.367854] Key type ._llcrypt registered [ 2286.369448] Key type .llcrypt registered [ 2286.413781] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 2286.776395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2286.790624] alg: No test for adler32 (adler32-zlib) [ 2287.699727] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 2287.804232] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 2289.411723] Key type lgssc registered [ 2289.918276] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2293.344526] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2293.353760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2294.598216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2296.343933] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2300.449471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2300.645978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2302.399713] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2304.530324] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2305.963741] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2308.651058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2308.807510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2311.237329] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2313.545654] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2320.104381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 2328.225722] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2329.732070] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2329.734284] Lustre: Skipped 2 previous similar messages [ 2331.541278] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2331.543872] Lustre: Skipped 2 previous similar messages [ 2335.759744] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2335.779549] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2341.440197] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.58@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2341.444216] Lustre: Skipped 2 previous similar messages [ 2342.533697] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.58@tcp 192.168.0.[2,10]@tcp [ 2347.815825] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.58/24@tcp in log lustre-MDT0000 [ 2347.818521] Lustre: Skipped 2 previous similar messages [ 2351.241956] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.58/24@tcp [ 2351.245097] Lustre: Skipped 1 previous similar message [ 2356.195832] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2356.196832] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2356.197633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2356.204956] Lustre: Skipped 1 previous similar message [ 2360.031947] Lustre: server umount lustre-OST0000 complete [ 2362.336258] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2362.338785] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2362.343365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2362.345608] Lustre: Skipped 1 previous similar message [ 2366.432340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2366.432909] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2367.642274] Lustre: server umount lustre-MDT0000 complete [ 2369.128150] LustreError: 115564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772475623 with bad export cookie 3967759839513287734 [ 2369.130483] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2369.132309] LustreError: 115564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2369.266520] Lustre: server umount lustre-MDT0001 complete [ 2373.482229] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 2374.817738] Key type lgssc unregistered [ 2374.968336] LNet: 118500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2374.972488] LNetError: 118500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2374.984448] LNet: Removed LNI 192.168.204.158@tcp [ 2375.361785] Key type .llcrypt unregistered [ 2375.363447] Key type ._llcrypt unregistered [ 2387.472092] Key type ._llcrypt registered [ 2387.473222] Key type .llcrypt registered [ 2387.521292] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 2388.022159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2388.032064] alg: No test for adler32 (adler32-zlib) [ 2388.909846] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 2389.022540] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 2390.623369] Key type lgssc registered [ 2391.063086] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2394.061647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2394.068204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2395.244327] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2395.255267] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.58/24@tcp [ 2395.257423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2396.742262] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2399.914024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2400.019973] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2400.029869] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.58/24@tcp [ 2400.047029] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2401.379582] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2403.398340] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2404.800923] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2407.387707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2407.518496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2409.655395] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2411.797051] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2420.839647] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 2426.339639] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2426.340498] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2426.340623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2426.347665] Lustre: Skipped 1 previous similar message [ 2427.747997] Lustre: server umount lustre-OST0000 complete [ 2430.943838] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2430.947281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2430.952727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2430.955191] Lustre: Skipped 1 previous similar message [ 2431.384310] Lustre: server umount lustre-MDT0000 complete [ 2431.456942] LustreError: 120020:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2431.465112] LustreError: 120020:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2432.815327] LustreError: 120001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772475686 with bad export cookie 1117410845891549083 [ 2432.818139] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2432.821647] LustreError: 120001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2432.970297] Lustre: server umount lustre-MDT0001 complete [ 2437.269517] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 2438.393687] Key type lgssc unregistered [ 2438.534838] LNet: 123272:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2438.538213] LNetError: 123272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2438.554665] LNet: Removed LNI 192.168.204.158@tcp [ 2438.891062] Key type .llcrypt unregistered [ 2438.892614] Key type ._llcrypt unregistered [ 2449.052811] Key type ._llcrypt registered [ 2449.054325] Key type .llcrypt registered [ 2449.104370] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 2456.244112] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 2456.763444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2456.784297] alg: No test for adler32 (adler32-zlib) [ 2457.656877] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 2457.751235] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 2459.343145] Key type lgssc registered [ 2459.799114] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2473.524205] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2473.534053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2474.643577] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2474.660799] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2474.701721] Lustre: lustre-MDT0000: new disk, initializing [ 2474.736808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2474.743461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2476.383817] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2481.302949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2481.355412] Lustre: 126609:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2481.384594] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2481.386482] Lustre: Skipped 1 previous similar message [ 2481.425513] Lustre: lustre-MDT0001: new disk, initializing [ 2481.473719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2481.496669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2481.502527] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2483.245178] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2486.734232] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2488.027653] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2490.549113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2490.657910] Lustre: lustre-OST0000: new disk, initializing [ 2490.660079] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2490.693761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2492.830270] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2493.201472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2493.207538] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2493.237697] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2496.244223] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2498.038175] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2498.126973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2499.566410] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2499.641693] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2503.652027] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2503.652888] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2503.658138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2503.663357] Lustre: Skipped 1 previous similar message [ 2506.466571] Lustre: server umount lustre-OST0000 complete [ 2508.768685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2508.770840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2508.773285] Lustre: Skipped 1 previous similar message [ 2508.775545] Lustre: Skipped 1 previous similar message [ 2513.888799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2513.892501] Lustre: Skipped 2 previous similar messages [ 2514.030296] Lustre: server umount lustre-MDT0000 complete [ 2515.556220] LustreError: 127385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772475769 with bad export cookie 1301839625145172620 [ 2515.558905] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2515.560862] LustreError: 127385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2515.689214] Lustre: server umount lustre-MDT0001 complete [ 2519.133068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2519.339882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2521.085836] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2524.365348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2524.492869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2525.929433] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2528.033410] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2529.352361] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2531.941652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2534.230921] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2536.293974] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2537.856318] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2545.091368] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 2546.565658] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2546.644382] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2550.315028] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2550.376662] Lustre: lustre-OST0001: new disk, initializing [ 2550.378808] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2550.407664] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2550.410577] Lustre: Skipped 1 previous similar message [ 2552.467325] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2555.895203] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2560.007539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2560.010875] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2560.049840] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2561.077536] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 2562.568750] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2562.653688] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2564.074086] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d8648605800.ost_server_uuid 50 [ 2564.718867] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d8648605800.ost_server_uuid in FULL state after 0 sec [ 2570.217754] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2570.220844] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2570.226554] LustreError: Skipped 2 previous similar messages [ 2570.233944] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2570.238793] Lustre: Skipped 1 previous similar message [ 2573.382332] Lustre: server umount lustre-OST0001 complete [ 2575.328411] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2575.330179] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2575.330203] LustreError: 134183:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2575.330214] LustreError: 134183:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2575.333558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2575.338022] Lustre: Skipped 2 previous similar messages [ 2580.450947] LustreError: 132007:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2580.458292] LustreError: 132007:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2581.095604] Lustre: server umount lustre-OST0000 complete [ 2585.569666] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2585.575948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2585.578421] Lustre: Skipped 4 previous similar messages [ 2588.699745] Lustre: server umount lustre-MDT0000 complete [ 2590.040490] LustreError: 133463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772475843 with bad export cookie 1301839625145173509 [ 2590.044089] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2590.045226] LustreError: 133463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2590.166604] Lustre: server umount lustre-MDT0001 complete [ 2596.631045] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 2597.667332] Key type lgssc unregistered [ 2597.797380] LNet: 136334:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2597.799700] LNetError: 136334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2597.811284] LNet: Removed LNI 192.168.204.158@tcp [ 2598.102335] Key type .llcrypt unregistered [ 2598.104013] Key type ._llcrypt unregistered [ 2611.291838] Key type ._llcrypt registered [ 2611.292726] Key type .llcrypt registered [ 2611.332971] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 2617.043978] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 2617.521344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2617.541791] alg: No test for adler32 (adler32-zlib) [ 2618.408530] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 2618.496786] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 2620.087182] Key type lgssc registered [ 2620.473246] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2632.650056] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2632.655529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2633.751046] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2633.761604] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2633.798079] Lustre: lustre-MDT0000: new disk, initializing [ 2633.825707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2633.833583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2635.201285] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2639.278286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2639.319136] Lustre: 140427:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2639.343716] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2639.346163] Lustre: Skipped 1 previous similar message [ 2639.382065] Lustre: lustre-MDT0001: new disk, initializing [ 2639.419742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2639.437985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2639.441965] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2640.805950] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2643.950307] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2645.176986] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2647.634278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2647.733964] Lustre: lustre-OST0000: new disk, initializing [ 2647.735983] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2647.764321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2649.656614] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2652.669527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2652.673939] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2652.711885] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2652.800171] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2654.840645] Lustre: Failing over lustre-OST0000 [ 2654.916201] Lustre: server umount lustre-OST0000 complete [ 2668.022573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2668.136885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2668.144748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2668.151919] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2669.673499] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2670.262253] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2671.832106] Lustre: Failing over lustre-MDT0000 [ 2671.924637] Lustre: server umount lustre-MDT0000 complete [ 2673.633423] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2673.638151] Lustre: Skipped 3 previous similar messages [ 2674.655134] Lustre: 138653:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772475911/real 1772475911] req@000000001453337c x1858575659983360/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1772475927 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2674.662316] Lustre: 138653:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2678.751130] Lustre: 138653:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772475916/real 1772475916] req@000000005f8d1aa3 x1858575659985408/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1772475932 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2678.752363] LustreError: 143948:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2678.762346] LustreError: 143948:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2679.904688] LustreError: 140434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2683.871095] Lustre: 138654:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772475921/real 1772475921] req@000000007e1e5cd3 x1858575659986432/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1772475937 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2683.871913] LustreError: 140438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2683.881513] LustreError: 140438:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2684.942778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2684.988349] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2684.992876] Lustre: 140429:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.204.158@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2685.024699] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.58@tcp (not set up) [ 2685.060853] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2685.079393] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2685.079767] Lustre: lustre-OST0000: Recovery over after 0:16, of 3 clients 3 recovered and 0 were evicted. [ 2685.079864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2685.107911] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2686.437897] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2690.145261] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2690.533028] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2690.534706] Lustre: Skipped 1 previous similar message [ 2690.539387] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2690.554990] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2695.650389] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2695.651931] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2695.651954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2695.654063] Lustre: Skipped 1 previous similar message [ 2697.605664] Lustre: server umount lustre-OST0000 complete [ 2700.768694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2700.768824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2700.770884] LustreError: Skipped 1 previous similar message [ 2700.781648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2700.783635] Lustre: Skipped 2 previous similar messages [ 2705.888183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2705.891036] Lustre: Skipped 1 previous similar message [ 2711.008156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2711.010568] Lustre: Skipped 1 previous similar message [ 2713.055113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2713.116437] Lustre: server umount lustre-MDT0000 complete [ 2714.465353] LustreError: 140418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772475968 with bad export cookie 13276701037727971318 [ 2714.468342] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2714.470158] LustreError: 140418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2714.593182] Lustre: server umount lustre-MDT0001 complete [ 2718.619754] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 2719.695740] Key type lgssc unregistered [ 2719.824506] LNet: 145653:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2719.827535] LNetError: 145653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2719.838370] LNet: Removed LNI 192.168.204.158@tcp [ 2720.154057] Key type .llcrypt unregistered [ 2720.155125] Key type ._llcrypt unregistered [ 2740.343433] Key type ._llcrypt registered [ 2740.344501] Key type .llcrypt registered [ 2740.378740] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 2740.816667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2740.828646] alg: No test for adler32 (adler32-zlib) [ 2741.702337] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 2741.792739] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 2743.375137] Key type lgssc registered [ 2743.721493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2746.293290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2746.300086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2747.461941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2748.745646] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2751.931642] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2754.035457] LustreError: 147183:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2754.038822] LustreError: 147183:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2754.041469] LustreError: 147183:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 2754.156965] Lustre: server umount lustre-MDT0000 complete [ 2759.039451] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 2762.159712] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 2774.433946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2774.526950] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2774.539676] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2774.576190] Lustre: lustre-MDT0000: new disk, initializing [ 2774.609758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2774.617730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2775.969981] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2780.318277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2780.359941] Lustre: 150333:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2780.383579] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2780.386340] Lustre: Skipped 1 previous similar message [ 2780.425175] Lustre: lustre-MDT0001: new disk, initializing [ 2780.457588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2780.471977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2780.475738] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2781.798697] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2784.889641] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2786.031278] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2788.161567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2788.273603] Lustre: lustre-OST0000: new disk, initializing [ 2788.275818] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2788.307798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2790.128708] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2791.543973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2791.546626] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2791.574611] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2793.111103] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2794.919347] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2795.457666] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 13:27:27 (1772476047) [ 2858.463450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2858.463857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2858.467294] Lustre: Skipped 2 previous similar messages [ 2858.468695] Lustre: Skipped 1 previous similar message [ 2863.584511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2863.586452] Lustre: Skipped 3 previous similar messages [ 2868.703589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2868.705777] Lustre: Skipped 2 previous similar messages [ 2872.287126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2872.365274] Lustre: server umount lustre-MDT0000 complete [ 2873.651668] LustreError: 150324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772476127 with bad export cookie 13100226484396607065 [ 2873.652763] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2873.655670] LustreError: 150324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2873.823478] LustreError: 150340:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2873.823839] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2873.828752] LustreError: 150340:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2873.832607] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2878.943578] LustreError: 150341:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2878.943582] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2879.776361] Lustre: server umount lustre-MDT0001 complete [ 2887.247869] Lustre: server umount lustre-OST0000 complete [ 2890.346793] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 2893.329173] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 2905.524744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2905.631169] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2905.643773] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2905.686233] Lustre: lustre-MDT0000: new disk, initializing [ 2905.712573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2905.719681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2907.068748] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2911.477249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2911.523343] Lustre: 156551:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2911.546265] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2911.547946] Lustre: Skipped 1 previous similar message [ 2911.586020] Lustre: lustre-MDT0001: new disk, initializing [ 2911.651165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2911.654794] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2913.018441] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2916.066605] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2917.269205] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2919.523200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2919.635548] Lustre: lustre-OST0000: new disk, initializing [ 2919.637992] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2919.669677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2919.672644] Lustre: Skipped 1 previous similar message [ 2919.854184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2919.857856] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2919.894069] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2921.617655] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2924.647025] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2925.940537] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2926.021695] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2927.318587] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2927.388815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2930.145533] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2930.146692] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2930.148097] LustreError: Skipped 1 previous similar message [ 2930.148668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2930.151486] Lustre: Skipped 1 previous similar message [ 2934.160651] Lustre: server umount lustre-OST0000 complete [ 2937.311963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2937.314620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2941.532424] Lustre: server umount lustre-MDT0000 complete [ 2942.710204] LustreError: 156543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772476196 with bad export cookie 13100226484396765881 [ 2942.713024] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2942.713781] LustreError: 156543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2942.827461] Lustre: server umount lustre-MDT0001 complete [ 2945.606918] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 13:29:58 (1772476198) [ 2950.718994] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 2953.656239] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 2964.955468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2965.048693] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2965.061508] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2965.097855] Lustre: lustre-MDT0000: new disk, initializing [ 2965.120037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2965.125806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2966.364776] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2970.435734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2970.478600] Lustre: 162862:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2970.570716] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2971.887195] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2975.016386] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2976.164779] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2978.402846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2979.016668] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2979.050559] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2980.349911] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2983.366513] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2989.537435] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2989.537894] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2989.539393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2989.539400] Lustre: Skipped 4 previous similar messages [ 2989.540300] LustreError: Skipped 1 previous similar message [ 2989.543877] Lustre: Skipped 2 previous similar messages [ 2991.309988] Lustre: server umount lustre-OST0000 complete [ 2998.741862] Lustre: server umount lustre-MDT0000 complete [ 2999.776134] LustreError: 162872:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2999.783127] LustreError: 162872:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3000.058133] LustreError: 163643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772476253 with bad export cookie 13100226484396766749 [ 3000.060765] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3000.061254] LustreError: 163643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3000.178138] Lustre: server umount lustre-MDT0001 complete [ 3003.169856] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 13:30:55 (1772476255) [ 3008.578028] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 3011.765463] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 3023.612611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3023.700911] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3023.711614] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3023.712961] Lustre: Skipped 3 previous similar messages [ 3023.747154] Lustre: lustre-MDT0000: new disk, initializing [ 3023.748385] Lustre: Skipped 2 previous similar messages [ 3023.773787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3023.777146] Lustre: Skipped 2 previous similar messages [ 3023.784706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3023.787207] Lustre: Skipped 2 previous similar messages [ 3025.049246] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3029.139694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3029.180290] Lustre: 168777:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3029.287667] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3030.637447] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3033.815305] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3035.072188] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3037.317088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3039.330296] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3039.420159] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3042.565794] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3044.832113] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3044.834839] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3044.838384] Lustre: Skipped 2 previous similar messages [ 3044.840353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3044.842237] Lustre: Skipped 3 previous similar messages [ 3049.441371] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3050.509600] Lustre: server umount lustre-OST0000 complete [ 3066.335425] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3066.412954] Lustre: server umount lustre-MDT0000 complete [ 3067.626259] LustreError: 168768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772476321 with bad export cookie 13100226484396767820 [ 3067.628842] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3067.630235] LustreError: 168768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3067.747827] Lustre: server umount lustre-MDT0001 complete [ 3071.181044] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 3072.177511] Key type lgssc unregistered [ 3072.314538] LNet: 172253:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3072.316812] LNetError: 172253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3072.323298] LNet: Removed LNI 192.168.204.158@tcp [ 3072.592492] Key type .llcrypt unregistered [ 3072.593654] Key type ._llcrypt unregistered [ 3081.313887] Key type ._llcrypt registered [ 3081.315442] Key type .llcrypt registered [ 3081.352893] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 3081.778554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3081.790746] alg: No test for adler32 (adler32-zlib) [ 3082.632268] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 3082.704446] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 3084.279188] Key type lgssc registered [ 3084.656277] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3087.121971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3087.128388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3088.302034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3089.544758] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3092.413090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3092.554428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3093.873548] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3095.601180] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3096.696383] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3098.813824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3098.941892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3100.748840] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3102.476887] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3113.318096] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3133.920926] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3133.921993] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3133.922620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3133.924649] Lustre: Skipped 1 previous similar message [ 3133.926747] LustreError: Skipped 1 previous similar message [ 3135.756743] Lustre: server umount lustre-OST0000 complete [ 3138.527819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3138.529728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3138.533075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3138.534856] Lustre: Skipped 1 previous similar message [ 3144.160107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3144.161997] Lustre: Skipped 3 previous similar messages [ 3149.280102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3149.282552] Lustre: Skipped 1 previous similar message [ 3151.327155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3151.381132] Lustre: server umount lustre-MDT0000 complete [ 3152.601015] LustreError: 173609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772476406 with bad export cookie 11565298731012971658 [ 3152.603678] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3152.604759] LustreError: 173609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3152.714255] Lustre: server umount lustre-MDT0001 complete [ 3156.181842] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 3157.163370] Key type lgssc unregistered [ 3157.283541] LNet: 176829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3157.286629] LNetError: 176829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3157.294441] LNet: Removed LNI 192.168.204.158@tcp [ 3157.593074] Key type .llcrypt unregistered [ 3157.594216] Key type ._llcrypt unregistered [ 3166.277747] Key type ._llcrypt registered [ 3166.279216] Key type .llcrypt registered [ 3166.317685] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 3166.607605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3166.646027] alg: No test for adler32 (adler32-zlib) [ 3167.506105] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 3167.588409] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 3169.167124] Key type lgssc registered [ 3169.499972] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3171.972433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3171.979397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3173.129905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3174.322503] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3176.986808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3177.117122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3178.379387] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3180.172892] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3181.383791] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3183.640878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3183.742504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3185.470790] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3187.084747] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3198.117251] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3207.647932] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3207.650051] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3207.654094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3208.672322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3208.672975] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3208.673468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3208.673824] Lustre: Skipped 1 previous similar message [ 3210.959671] Lustre: server umount lustre-OST0000 complete [ 3212.653712] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8d864743e000.ost_server_uuid 50 [ 3213.161640] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8d864743e000.ost_server_uuid in DISCONN state after 0 sec [ 3213.699348] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3223.738396] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3223.743519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3223.747388] Lustre: Skipped 1 previous similar message [ 3228.639896] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3228.642477] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3228.646968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3229.906295] Lustre: server umount lustre-MDT0000 complete [ 3231.107833] LustreError: 178922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772476484 with bad export cookie 11439043905346617490 [ 3231.110776] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3231.111814] LustreError: 178922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3231.208764] Lustre: server umount lustre-MDT0001 complete [ 3233.837490] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 13:34:46 (1772476486) [ 3235.937485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3236.074739] LustreError: 181736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3236.080015] LustreError: 181736:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3236.096920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3237.327626] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3239.977320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3241.228353] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3242.874966] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3244.040949] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3246.223188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3246.333759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3246.335541] Lustre: Skipped 1 previous similar message [ 3248.111536] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3249.781880] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3251.920593] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3251.990334] Lustre: lustre-OST0001: new disk, initializing [ 3251.992195] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3253.711705] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3256.644962] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3257.340503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3257.343518] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3257.369339] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3260.455294] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3264.991722] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3264.994306] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3264.997911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3264.999178] Lustre: Skipped 2 previous similar messages [ 3268.793841] Lustre: server umount lustre-OST0000 complete [ 3270.628106] LustreError: 183331:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3270.632988] LustreError: 183331:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3270.755699] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3270.847611] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3272.234896] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3272.289847] LustreError: 184846:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3272.308822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3275.744861] LustreError: 183331:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3275.750637] LustreError: 183331:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3280.868916] LustreError: 184844:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3280.874648] LustreError: 184844:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3282.912214] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3282.912373] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3282.914346] LustreError: Skipped 2 previous similar messages [ 3282.917767] Lustre: Skipped 1 previous similar message [ 3282.919737] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3282.923495] Lustre: Skipped 4 previous similar messages [ 3288.529474] Lustre: server umount lustre-OST0001 complete [ 3296.028202] Lustre: server umount lustre-MDT0000 complete [ 3296.224277] LustreError: 181737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3296.229446] LustreError: 181737:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3297.305081] LustreError: 184237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772476551 with bad export cookie 11439043905346618547 [ 3297.307676] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3297.308800] LustreError: 184237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3297.416636] Lustre: server umount lustre-MDT0001 complete [ 3304.760617] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 13:35:57 (1772476557) [ 3306.770522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3306.856938] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3306.868647] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3306.940865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3306.943600] Lustre: Skipped 1 previous similar message [ 3308.122248] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3310.699187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3310.732459] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3310.744852] Lustre: 187508:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3312.169328] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3314.037609] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3315.252943] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3317.458928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3317.525852] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3319.378163] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 3319.432568] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3321.206319] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3323.541905] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3323.566956] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3323.650061] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3323.652994] Lustre: Skipped 2 previous similar messages [ 3324.639091] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 3325.597552] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3327.372223] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3329.504077] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3329.504336] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3329.506538] LustreError: Skipped 1 previous similar message [ 3329.506982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3329.511536] Lustre: Skipped 3 previous similar messages [ 3329.513535] Lustre: Skipped 4 previous similar messages [ 3334.205830] Lustre: server umount lustre-OST0000 complete [ 3335.138757] LustreError: 189463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3335.142543] LustreError: 189463:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3350.796231] Lustre: server umount lustre-OST0001 complete [ 3352.031864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3352.033896] LustreError: Skipped 2 previous similar messages [ 3365.856201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3365.858046] Lustre: Skipped 14 previous similar messages [ 3366.367164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3366.437967] Lustre: server umount lustre-MDT0000 complete [ 3367.669945] LustreError: 187498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772476621 with bad export cookie 11439043905346619933 [ 3367.672178] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3367.674355] LustreError: 187498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3367.791918] Lustre: server umount lustre-MDT0001 complete [ 3374.750493] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 13:37:07 (1772476627) [ 3379.468723] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 3382.090760] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 3392.621408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3392.699870] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3392.710245] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3392.743338] Lustre: lustre-MDT0000: new disk, initializing [ 3392.764412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3392.770211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3393.952894] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3397.744635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3397.777825] Lustre: 195081:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3397.797947] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3397.799590] Lustre: Skipped 1 previous similar message [ 3397.836081] Lustre: lustre-MDT0001: new disk, initializing [ 3397.876206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3397.880740] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3399.019762] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3401.878421] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3402.972705] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3405.035504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3405.123357] Lustre: lustre-OST0000: new disk, initializing [ 3405.125198] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3406.783194] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3409.634564] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3409.907991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3409.912739] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3409.937016] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3410.821937] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3410.890263] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3412.028196] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3412.087575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3415.010578] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3415.010606] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3415.013977] LustreError: Skipped 1 previous similar message [ 3415.019034] Lustre: Skipped 5 previous similar messages [ 3418.771439] Lustre: server umount lustre-OST0000 complete [ 3426.128403] Lustre: server umount lustre-MDT0000 complete [ 3427.271995] LustreError: 195071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772476681 with bad export cookie 11439043905346621704 [ 3427.274905] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3427.278255] LustreError: 195071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3427.371196] Lustre: server umount lustre-MDT0001 complete [ 3430.102496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3430.249659] LustreError: 198887:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3430.255080] LustreError: 198887:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 3431.433721] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3434.043818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3435.322641] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3436.920477] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3438.050505] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3440.138484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3441.893748] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3443.585042] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3444.935948] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3445.012364] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3446.365424] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3446.434633] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3449.827230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3449.829842] Lustre: Skipped 7 previous similar messages [ 3453.197895] Lustre: server umount lustre-OST0000 complete [ 3455.204903] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3456.583364] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3480.195856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3480.312477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3480.315459] Lustre: Skipped 5 previous similar messages [ 3482.166698] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3483.906934] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3484.463681] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3487.507067] Lustre: server umount lustre-OST0000 complete [ 3488.740476] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3488.745112] Lustre: Skipped 5 previous similar messages [ 3494.874613] Lustre: server umount lustre-MDT0000 complete [ 3494.880725] LustreError: 200363:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3494.885320] LustreError: 200363:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3496.209410] LustreError: 198868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772476750 with bad export cookie 11439043905346622572 [ 3496.213400] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3496.215284] LustreError: 198868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3496.332761] Lustre: server umount lustre-MDT0001 complete [ 3499.167251] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 13:39:11 (1772476751) [ 3501.377224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3502.930232] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3505.877223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3507.213135] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3508.909444] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3510.049326] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3512.202558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3514.156849] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3515.832550] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3517.111731] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3527.368376] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 3528.618820] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3528.684576] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3530.825438] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3530.887780] Lustre: lustre-OST0001: new disk, initializing [ 3530.889923] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3532.705433] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3535.656811] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3536.985164] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3540.986369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3540.989052] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3541.010925] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3541.119986] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 3542.311480] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3542.385590] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3546.080918] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3546.082928] LustreError: Skipped 1 previous similar message [ 3549.053684] Lustre: server umount lustre-OST0001 complete [ 3550.840885] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3550.914787] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3552.128284] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3552.198619] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3575.574618] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3577.380788] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3579.064040] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3579.602418] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3580.899559] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3580.900900] Lustre: Skipped 7 previous similar messages [ 3586.299200] Lustre: server umount lustre-OST0001 complete [ 3604.435591] Lustre: server umount lustre-OST0000 complete [ 3611.927968] Lustre: server umount lustre-MDT0000 complete [ 3613.157659] LustreError: 204352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772476867 with bad export cookie 11439043905346623489 [ 3613.159151] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3613.161703] LustreError: 204352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3619.309226] Lustre: server umount lustre-MDT0001 complete [ 3626.664774] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 13:41:19 (1772476879) [ 3628.977928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3629.089971] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3629.103366] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3629.179947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3629.182128] Lustre: Skipped 5 previous similar messages [ 3630.375668] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3633.219411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3633.252142] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3633.262203] Lustre: 212368:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3634.708478] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3636.444730] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3637.635506] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3639.848787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3639.917167] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3641.822624] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3643.566447] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3647.282207] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3649.288921] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3651.132431] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3654.755282] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 3656.078571] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3656.155188] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3657.445160] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3657.513852] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3658.709877] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d8644e2b000.ost_server_uuid 50 [ 3659.213085] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d8644e2b000.ost_server_uuid in FULL state after 0 sec [ 3659.554304] Lustre: Permanently deactivating lustre-OST0001 [ 3659.559741] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3660.374894] Lustre: Permanently reactivating lustre-OST0001 [ 3662.818825] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3662.824272] Lustre: Skipped 7 previous similar messages [ 3667.449501] Lustre: server umount lustre-OST0001 complete [ 3667.941820] LustreError: 213982:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3667.946224] LustreError: 213982:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 23 previous similar messages [ 3674.770157] Lustre: server umount lustre-OST0000 complete [ 3682.137918] Lustre: server umount lustre-MDT0000 complete [ 3683.428244] LustreError: 214894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772476937 with bad export cookie 11439043905346624749 [ 3683.431980] LustreError: 214894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3683.432617] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3683.546262] Lustre: server umount lustre-MDT0001 complete [ 3691.272838] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 13:42:23 (1772476943) [ 3693.744709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3693.828181] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3693.839436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3693.842741] Lustre: Skipped 5 previous similar messages [ 3695.055660] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3697.756590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3697.790358] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3697.792021] Lustre: Skipped 1 previous similar message [ 3697.800454] Lustre: 218466:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3699.167225] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3701.010465] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3702.222326] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3704.518024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3704.600396] Lustre: Permanently deactivating lustre-OST0000 [ 3704.606588] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3706.532732] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3708.370699] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3710.446482] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3710.674733] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3712.667526] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3714.521580] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3717.139453] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3720.181607] Lustre: Permanently reactivating lustre-OST0000 [ 3724.791052] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3724.794054] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3724.798979] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3724.816421] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3724.822673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3730.915401] LustreError: lustre-OST0001-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 3730.917747] LustreError: Skipped 5 previous similar messages [ 3734.206248] Lustre: server umount lustre-OST0001 complete [ 3737.551282] Lustre: server umount lustre-OST0000 complete [ 3748.862991] Lustre: server umount lustre-MDT0000 complete [ 3750.104125] LustreError: 218459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772477003 with bad export cookie 11439043905346626380 [ 3750.107717] LustreError: 218459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3750.107925] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3759.632368] Lustre: server umount lustre-MDT0001 complete [ 3763.071915] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 3764.098244] Key type lgssc unregistered [ 3764.225329] LNet: 222850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3764.228053] LNetError: 222850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3764.238266] LNet: Removed LNI 192.168.204.158@tcp [ 3764.557069] Key type .llcrypt unregistered [ 3764.558910] Key type ._llcrypt unregistered [ 3773.275818] Key type ._llcrypt registered [ 3773.276809] Key type .llcrypt registered [ 3773.313463] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 3773.702302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3773.734058] alg: No test for adler32 (adler32-zlib) [ 3774.582926] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 3774.666266] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 3776.247133] Key type lgssc registered [ 3776.633659] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3780.033785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3780.039723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3781.195276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3782.534990] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3785.393222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3785.431678] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3785.444066] Lustre: 224265:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3785.448438] Lustre: Permanently deactivating lustre-MDT0001 [ 3785.454476] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3785.569777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3786.925417] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3788.695682] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3789.292856] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3789.295080] LustreError: 224305:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3789.298673] LustreError: 224305:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3789.303763] LustreError: 224305:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 1, failed: rc = -108 [ 3789.898988] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3792.178516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3792.297826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3794.174512] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3795.363304] LustreError: 226392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3795.367264] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3795.369293] LustreError: 226392:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3796.040385] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3798.475484] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3798.545063] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3800.524093] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3802.326372] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3803.622555] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3806.696630] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3806.696748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3808.316573] Lustre: Permanently reactivating lustre-MDT0001 [ 3808.322518] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3808.324993] Lustre: Skipped 2 previous similar messages [ 3809.512713] Lustre: lustre-MDT0001: Client 7aecd86d-3187-4c05-8cdf-d7ee7bc8a0ed (at 192.168.204.58@tcp) reconnecting [ 3811.639957] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3812.161105] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3813.638901] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3813.705150] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3814.930371] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3815.009388] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3815.811249] Lustre: Permanently deactivating lustre-MDT0001 [ 3815.817267] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3815.820768] Lustre: Skipped 2 previous similar messages [ 3819.903681] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3824.565751] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 3824.569499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3824.575885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3827.168862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3827.170812] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3827.171661] Lustre: Skipped 1 previous similar message [ 3827.176122] Lustre: Skipped 1 previous similar message [ 3830.744372] Lustre: server umount lustre-MDT0000 complete [ 3832.044030] LustreError: 227372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772477085 with bad export cookie 7621970279705967468 [ 3832.046993] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3832.047622] LustreError: 227372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3832.162427] Lustre: server umount lustre-MDT0001 complete [ 3839.484793] Lustre: server umount lustre-OST0000 complete [ 3843.330802] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 13:44:55 (1772477095) [ 3846.863152] Lustre: server umount lustre-OST0001 complete [ 3848.906462] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 3852.196106] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 3864.123101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3864.211577] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3864.214470] Lustre: Skipped 2 previous similar messages [ 3864.222967] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3864.256110] Lustre: lustre-MDT0000: new disk, initializing [ 3864.277846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3864.283731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3865.503465] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3869.662670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3869.704367] Lustre: 231956:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3869.727520] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3869.729265] Lustre: Skipped 1 previous similar message [ 3869.771425] Lustre: lustre-MDT0001: new disk, initializing [ 3869.813627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3869.816897] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3871.046268] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3874.077356] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3875.232692] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3877.368234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3877.457528] Lustre: lustre-OST0000: new disk, initializing [ 3877.459182] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3877.482551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3877.484483] Lustre: Skipped 1 previous similar message [ 3879.187992] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3882.163793] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3883.509064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3883.512062] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3883.541355] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3888.187724] LustreError: 231964:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3892.455971] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3892.520092] Lustre: lustre-OST0001: new disk, initializing [ 3892.521950] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3894.419673] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3897.551818] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3898.106489] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3898.279125] LustreError: 231964:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3898.287095] LustreError: 233194:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3900.926237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3900.929521] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3900.955872] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3908.375067] LustreError: 233194:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3911.140821] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3911.141589] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3911.142796] LustreError: Skipped 1 previous similar message [ 3911.146807] Lustre: Skipped 1 previous similar message [ 3911.149202] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3911.151905] Lustre: Skipped 1 previous similar message [ 3915.263928] Lustre: server umount lustre-OST0001 complete [ 3916.259896] LustreError: 233681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3916.264253] LustreError: 233681:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3921.378759] LustreError: 233681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3921.383037] LustreError: 233681:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3926.502032] LustreError: 233682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3926.506668] LustreError: 233682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3929.568556] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3929.568734] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3929.571293] LustreError: Skipped 1 previous similar message [ 3929.571865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3929.574866] Lustre: Skipped 1 previous similar message [ 3929.579689] Lustre: Skipped 2 previous similar messages [ 3931.619852] LustreError: 233682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3931.628291] LustreError: 233682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3933.327600] Lustre: server umount lustre-OST0000 complete [ 3936.737241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3936.737408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3936.742175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3936.747781] Lustre: Skipped 3 previous similar messages [ 3940.826605] Lustre: server umount lustre-MDT0000 complete [ 3941.856018] LustreError: 231968:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3941.859963] LustreError: 231968:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3942.027965] LustreError: 231947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772477195 with bad export cookie 7621970279705970282 [ 3942.031046] LustreError: 231947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3942.031320] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3948.220945] Lustre: server umount lustre-MDT0001 complete [ 3951.701938] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 3952.684631] Key type lgssc unregistered [ 3952.810282] LNet: 236620:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3952.812372] LNetError: 236620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3952.821319] LNet: Removed LNI 192.168.204.158@tcp [ 3953.142750] Key type .llcrypt unregistered [ 3953.143736] Key type ._llcrypt unregistered [ 3967.658877] Key type ._llcrypt registered [ 3967.660181] Key type .llcrypt registered [ 3967.699301] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 3968.146267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3968.161110] alg: No test for adler32 (adler32-zlib) [ 3969.020868] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 3969.109384] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 3970.695126] Key type lgssc registered [ 3971.062415] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3973.687202] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3973.693192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3974.790420] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3974.799262] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3974.869126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3976.204646] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3979.018698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3979.050442] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3979.061040] Lustre: 238787:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3979.157940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3980.537619] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3982.297358] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3983.493417] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3985.716701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3985.789507] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3985.850610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3987.699980] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3989.548363] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3992.589787] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 4038.625547] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4038.626484] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4038.629052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4038.632590] Lustre: Skipped 1 previous similar message [ 4040.609937] Lustre: server umount lustre-OST0000 complete [ 4041.994632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4045.764017] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4045.879783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4047.516559] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4049.145537] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4058.151235] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4058.155514] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4058.155920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4083.681157] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4083.682912] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4083.683390] LustreError: Skipped 2 previous similar messages [ 4083.683882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4083.687933] Lustre: Skipped 1 previous similar message [ 4083.689853] Lustre: Skipped 1 previous similar message [ 4088.592265] Lustre: server umount lustre-OST0000 complete [ 4091.871677] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4091.874169] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4091.878420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4093.919872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4093.920573] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4099.040050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4099.042375] Lustre: Skipped 2 previous similar messages [ 4104.159130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4104.219965] Lustre: server umount lustre-MDT0000 complete [ 4105.380107] LustreError: 238777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772477359 with bad export cookie 12404545877512308034 [ 4105.383240] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4105.385625] LustreError: 238777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4105.505491] Lustre: server umount lustre-MDT0001 complete [ 4108.817577] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 4109.753629] Key type lgssc unregistered [ 4109.881337] LNet: 243736:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4109.883298] LNetError: 243736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4109.891260] LNet: Removed LNI 192.168.204.158@tcp [ 4110.149080] Key type .llcrypt unregistered [ 4110.150577] Key type ._llcrypt unregistered [ 4119.019264] Key type ._llcrypt registered [ 4119.020387] Key type .llcrypt registered [ 4119.058105] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 4119.373652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4119.393060] alg: No test for adler32 (adler32-zlib) [ 4120.247244] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 4120.325561] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 4121.911136] Key type lgssc registered [ 4122.252629] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4124.857292] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4124.863558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4126.041836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4127.289344] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4130.115273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4130.241491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4131.479806] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4133.220896] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4134.462734] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4136.711468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4136.850449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4138.709349] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4140.613055] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4150.183442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4160.481480] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4160.483146] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4160.485656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4160.490110] Lustre: Skipped 1 previous similar message [ 4160.494311] Lustre: Skipped 1 previous similar message [ 4165.011269] Lustre: server umount lustre-OST0000 complete [ 4170.720606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4170.721471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4170.724077] Lustre: Skipped 1 previous similar message [ 4172.440298] Lustre: server umount lustre-MDT0000 complete [ 4173.785431] LustreError: 245091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772477427 with bad export cookie 12735891095912719893 [ 4173.788819] LustreError: 245091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4173.788925] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4173.923069] Lustre: server umount lustre-MDT0001 complete [ 4177.748804] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 4178.822221] Key type lgssc unregistered [ 4178.958518] LNet: 249084:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4178.960686] LNetError: 249084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4178.972493] LNet: Removed LNI 192.168.204.158@tcp [ 4179.296723] Key type .llcrypt unregistered [ 4179.297786] Key type ._llcrypt unregistered [ 4186.646926] Key type ._llcrypt registered [ 4186.647942] Key type .llcrypt registered [ 4186.685210] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 4187.078866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4187.121033] alg: No test for adler32 (adler32-zlib) [ 4187.981199] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 4188.067212] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 4189.655162] Key type lgssc registered [ 4190.035279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4192.725114] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4192.733533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4193.890433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4195.140105] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4197.908054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4198.029878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4199.336124] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4201.176983] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4202.325723] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4204.582493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4204.697428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4206.556458] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4208.320115] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4219.046881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4228.576143] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4228.578376] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4228.584473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4229.600899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4229.601067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4229.601168] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4233.745335] Lustre: server umount lustre-OST0000 complete [ 4239.327925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4239.332917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4239.341851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4239.345176] Lustre: Skipped 1 previous similar message [ 4241.109945] Lustre: server umount lustre-MDT0000 complete [ 4242.317083] LustreError: 251880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772477496 with bad export cookie 13518967195050489358 [ 4242.320691] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4242.320691] LustreError: 251880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4242.435025] Lustre: server umount lustre-MDT0001 complete [ 4246.008564] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 4247.029894] Key type lgssc unregistered [ 4247.162454] LNet: 253609:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4247.164577] LNetError: 253609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4247.174348] LNet: Removed LNI 192.168.204.158@tcp [ 4247.500996] Key type .llcrypt unregistered [ 4247.502316] Key type ._llcrypt unregistered [ 4256.855584] Key type ._llcrypt registered [ 4256.857348] Key type .llcrypt registered [ 4256.903970] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 4257.371842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4257.398553] alg: No test for adler32 (adler32-zlib) [ 4258.252257] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 4258.333869] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 4259.911215] Key type lgssc registered [ 4260.264815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4262.758457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4262.765495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4263.929409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4265.245297] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4268.004055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4268.124649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4269.366134] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4271.050000] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4272.266468] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4274.463887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4274.589810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4276.387553] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4278.270225] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4288.934399] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4303.328072] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4303.331886] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4303.337705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4304.355654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4304.356344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4304.356930] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4304.367176] Lustre: Skipped 1 previous similar message [ 4308.015162] Lustre: server umount lustre-OST0000 complete [ 4314.079850] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4314.081733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4314.085057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4319.712486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4319.714574] Lustre: Skipped 2 previous similar messages [ 4324.319117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4324.376716] Lustre: server umount lustre-MDT0000 complete [ 4324.832485] LustreError: 254986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4324.838943] LustreError: 254986:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4325.630506] LustreError: 256599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772477579 with bad export cookie 11635331329845861724 [ 4325.635025] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4325.635401] LustreError: 256599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4325.760463] Lustre: server umount lustre-MDT0001 complete [ 4329.408551] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 4330.418710] Key type lgssc unregistered [ 4330.545451] LNet: 258958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4330.549671] LNetError: 258958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4330.559383] LNet: Removed LNI 192.168.204.158@tcp [ 4330.878238] Key type .llcrypt unregistered [ 4330.879394] Key type ._llcrypt unregistered [ 4337.861192] Key type ._llcrypt registered [ 4337.862098] Key type .llcrypt registered [ 4337.895636] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 4338.302508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4338.311268] alg: No test for adler32 (adler32-zlib) [ 4339.159982] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 4339.241703] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 4340.823186] Key type lgssc registered [ 4341.172605] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4343.718151] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4343.724301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4344.898521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4346.156591] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4348.940981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4349.068634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4350.315310] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4352.136775] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4353.398755] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4355.646976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4355.781939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4357.628147] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4359.400358] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4370.151208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4379.615827] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4379.617843] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4379.621194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4380.641832] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4380.642647] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4380.642730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4384.780809] Lustre: server umount lustre-OST0000 complete [ 4390.367931] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4390.370772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4390.374239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4390.376011] Lustre: Skipped 1 previous similar message [ 4392.154343] Lustre: server umount lustre-MDT0000 complete [ 4393.468395] LustreError: 260121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772477647 with bad export cookie 7707836681215460433 [ 4393.471356] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4393.471734] LustreError: 260121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4393.587996] Lustre: server umount lustre-MDT0001 complete [ 4397.072188] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 4398.027988] Key type lgssc unregistered [ 4398.143376] LNet: 263520:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4398.145420] LNetError: 263520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4398.155268] LNet: Removed LNI 192.168.204.158@tcp [ 4398.432501] Key type .llcrypt unregistered [ 4398.434311] Key type ._llcrypt unregistered [ 4401.890226] Key type ._llcrypt registered [ 4401.891052] Key type .llcrypt registered [ 4401.924358] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4407.130453] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 4413.070770] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 4413.394313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4413.401357] alg: No test for adler32 (adler32-zlib) [ 4414.253838] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 4414.334286] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 4415.919100] Key type lgssc registered [ 4416.295603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4427.685982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4427.690994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4428.779460] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4428.790418] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4428.827791] Lustre: lustre-MDT0000: new disk, initializing [ 4428.851602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4428.857632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4430.100038] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4434.193924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4434.232866] Lustre: 266992:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4434.252724] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4434.255071] Lustre: Skipped 1 previous similar message [ 4434.289514] Lustre: lustre-MDT0001: new disk, initializing [ 4434.321175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4434.335268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4434.339562] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4435.651640] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4438.664879] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4439.840281] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4442.047205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4442.146601] Lustre: lustre-OST0000: new disk, initializing [ 4442.148310] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4442.175239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4443.990707] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4446.711353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4446.714318] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4446.748039] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4447.036370] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4448.408708] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4448.476860] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4449.776071] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4449.841727] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4451.810857] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4451.811207] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4451.817530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4451.819083] LustreError: Skipped 1 previous similar message [ 4451.820816] Lustre: Skipped 1 previous similar message [ 4456.595277] Lustre: server umount lustre-OST0000 complete [ 4460.000021] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4460.002294] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4460.005640] Lustre: Skipped 1 previous similar message [ 4460.007420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4462.048227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4462.048774] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4467.168300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4467.170991] Lustre: Skipped 2 previous similar messages [ 4471.264640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4471.266539] Lustre: Skipped 1 previous similar message [ 4472.287077] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4472.350683] Lustre: server umount lustre-MDT0000 complete [ 4473.574703] LustreError: 266984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772477727 with bad export cookie 13534387902855851010 [ 4473.578533] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4473.579793] LustreError: 266984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4473.710977] Lustre: server umount lustre-MDT0001 complete [ 4476.371605] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 13:55:28 (1772477728) [ 4478.561610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4478.721507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4480.236703] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4483.497653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4483.630240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4485.009881] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4487.149996] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4488.335479] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4490.503461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4492.369743] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4494.094963] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4514.272433] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4514.275035] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4514.278627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4518.743182] Lustre: server umount lustre-OST0000 complete [ 4520.416559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4520.421116] Lustre: Skipped 2 previous similar messages [ 4526.301472] Lustre: server umount lustre-MDT0000 complete [ 4527.593387] LustreError: 270978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772477781 with bad export cookie 13534387902855851878 [ 4527.595539] LustreError: 271717:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4527.596021] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4527.598113] LustreError: 270978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4527.736565] Lustre: server umount lustre-MDT0001 complete [ 4531.217092] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 4532.302967] Key type lgssc unregistered [ 4532.440494] LNet: 274193:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4532.442973] LNetError: 274193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4532.451491] LNet: Removed LNI 192.168.204.158@tcp [ 4532.810720] Key type .llcrypt unregistered [ 4532.811837] Key type ._llcrypt unregistered [ 4541.332981] Key type ._llcrypt registered [ 4541.334039] Key type .llcrypt registered [ 4541.370388] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 4547.694916] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 4548.016925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4548.050452] alg: No test for adler32 (adler32-zlib) [ 4548.907489] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 4549.001919] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 4550.575110] Key type lgssc registered [ 4550.951060] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4565.753414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4565.758152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4566.846716] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4566.862040] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4566.899492] Lustre: lustre-MDT0000: new disk, initializing [ 4566.923156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4566.929227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4568.202689] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4572.522128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4572.560876] Lustre: 277958:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4572.581810] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4572.583441] Lustre: Skipped 1 previous similar message [ 4572.617758] Lustre: lustre-MDT0001: new disk, initializing [ 4572.647656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4572.660173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4572.662950] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4573.961524] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4577.036545] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4578.242750] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4580.507919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4580.616587] Lustre: lustre-OST2710: new disk, initializing [ 4580.620042] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4580.622485] Lustre: lustre-OST2710: Not available for connect from 0@lo (not set up) [ 4580.662171] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4582.532183] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4585.529455] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4585.963196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4585.966909] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4585.979185] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4587.817389] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4587.887806] Lustre: lustre-OST03e8: new disk, initializing [ 4587.889469] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4587.921578] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4589.796101] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4590.082231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4590.087887] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4590.118252] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4591.572371] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4594.083082] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4594.154519] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4595.469676] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4595.535631] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4596.807982] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4596.876741] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4598.177144] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4598.243653] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4600.291121] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4600.291935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4600.296414] Lustre: Skipped 1 previous similar message [ 4605.400188] Lustre: server umount lustre-MDT0000 complete [ 4606.576796] LustreError: 277948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772477860 with bad export cookie 14593527766951925659 [ 4606.581868] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4606.706904] Lustre: server umount lustre-MDT0001 complete [ 4617.728601] Lustre: server umount lustre-OST2710 complete [ 4629.149282] Lustre: server umount lustre-OST03e8 complete [ 4634.931293] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 4637.894078] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 4649.563754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4649.653790] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4649.663995] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4649.702199] Lustre: lustre-MDT0000: new disk, initializing [ 4649.726444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4649.733319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4651.071892] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4655.188888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4655.233930] Lustre: 285674:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4655.339154] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4656.607616] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4659.716834] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4660.922456] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4663.261113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4663.355350] Lustre: lustre-OST0000: new disk, initializing [ 4663.356888] Lustre: Skipped 1 previous similar message [ 4663.358506] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4663.360151] Lustre: Skipped 2 previous similar messages [ 4663.384770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4663.386853] Lustre: Skipped 1 previous similar message [ 4664.184601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4664.188158] Lustre: Skipped 1 previous similar message [ 4664.190953] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4664.238785] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4665.266381] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4668.337811] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4669.657536] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4669.720131] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4670.981606] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4671.050307] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4674.529543] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4674.530644] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4674.532449] LustreError: Skipped 1 previous similar message [ 4674.533058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4674.535895] Lustre: Skipped 3 previous similar messages [ 4674.536931] Lustre: Skipped 3 previous similar messages [ 4677.773627] Lustre: server umount lustre-OST0000 complete [ 4679.648719] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4679.649274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4679.652175] Lustre: Skipped 1 previous similar message [ 4679.654675] Lustre: Skipped 1 previous similar message [ 4684.768396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4684.770238] Lustre: Skipped 1 previous similar message [ 4685.085678] Lustre: server umount lustre-MDT0000 complete [ 4686.300089] LustreError: 285666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772477940 with bad export cookie 14593527766951927269 [ 4686.303257] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4686.305312] LustreError: 285666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4686.421527] Lustre: server umount lustre-MDT0001 complete [ 4689.108302] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 13:59:01 (1772477941) [ 4689.562248] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4690.088188] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 13:59:02 (1772477942) [ 4691.020701] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 4700.199834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4700.291450] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4700.298218] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4700.361148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4701.803964] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4704.919702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4704.955726] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4704.965631] Lustre: 291095:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4706.334638] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4708.033353] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4709.153407] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4722.426721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4722.502542] Lustre: Denying initial registration attempt from nid 192.168.204.158@tcp, specified as failover [ 4722.505239] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4722.509539] LustreError: 292706:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -99 [ 4722.513085] LustreError: 292706:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 4722.515587] LustreError: 292706:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4722.542103] Lustre: server umount lustre-OST0000 complete [ 4722.543295] LustreError: 292706:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4725.727875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4725.730817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4725.735387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4730.777824] Lustre: server umount lustre-MDT0000 complete [ 4731.872156] LustreError: 291108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4731.876863] LustreError: 291108:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 4731.925095] LustreError: 293076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772477985 with bad export cookie 14593527766951928081 [ 4731.927510] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4731.928315] LustreError: 293076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4732.035321] Lustre: server umount lustre-MDT0001 complete [ 4735.536917] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 4736.570182] Key type lgssc unregistered [ 4736.700438] LNet: 293810:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4736.702535] LNetError: 293810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4736.714369] LNet: Removed LNI 192.168.204.158@tcp [ 4737.007041] Key type .llcrypt unregistered [ 4737.008345] Key type ._llcrypt unregistered [ 4740.231821] Key type ._llcrypt registered [ 4740.232769] Key type .llcrypt registered [ 4740.267191] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 4740.568524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4740.575653] alg: No test for adler32 (adler32-zlib) [ 4741.419192] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 4741.497027] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 4743.071123] Key type lgssc registered [ 4743.408660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4778.278815] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 4781.353989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4781.360986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4782.469576] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4782.476780] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4782.539201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4783.897563] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4787.023690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4787.060791] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4787.074695] Lustre: 296534:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4787.190081] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4788.510281] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4790.321420] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4791.720493] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4805.483044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4805.557132] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4805.618722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4807.467871] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4809.284113] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4810.055425] Lustre: server umount lustre-OST0000 complete [ 4811.765335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4811.766984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4811.769238] Lustre: Skipped 1 previous similar message [ 4811.772330] Lustre: Skipped 1 previous similar message [ 4816.865187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4816.868424] Lustre: Skipped 1 previous similar message [ 4817.512255] Lustre: server umount lustre-MDT0000 complete [ 4818.948086] LustreError: 298172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772478072 with bad export cookie 12144799535297307979 [ 4818.951273] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4818.952793] LustreError: 298172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4819.104228] Lustre: server umount lustre-MDT0001 complete [ 4822.862495] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 4824.077349] Key type lgssc unregistered [ 4824.209391] LNet: 299751:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4824.211942] LNetError: 299751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4824.221329] LNet: Removed LNI 192.168.204.158@tcp [ 4824.538786] Key type .llcrypt unregistered [ 4824.540091] Key type ._llcrypt unregistered [ 4833.692194] Key type ._llcrypt registered [ 4833.693611] Key type .llcrypt registered [ 4833.740745] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 4834.125628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4834.169103] alg: No test for adler32 (adler32-zlib) [ 4835.042906] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 4835.135537] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 4836.719134] Key type lgssc registered [ 4837.075413] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4839.727828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4839.736200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4840.902243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4842.216296] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4845.023296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4845.155629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4846.518477] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4848.308363] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4849.453788] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4851.673406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4851.806609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4853.737167] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4855.525088] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4870.624043] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4870.626144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4870.630889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4871.272128] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.58@tcp (stopping) [ 4871.649713] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4871.653292] Lustre: Skipped 1 previous similar message [ 4874.272095] Lustre: server umount lustre-MDT0000 complete [ 4875.495322] LustreError: 301107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772478129 with bad export cookie 13168618014695479158 [ 4875.498761] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4875.501027] LustreError: 301107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4876.385432] LustreError: 301122:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4876.390521] LustreError: 301122:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4876.391047] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.58@tcp (stopping) [ 4876.396314] Lustre: Skipped 3 previous similar messages [ 4876.768422] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4880.799912] LustreError: 303281:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4880.800068] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4880.807049] LustreError: 303281:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4880.808627] Lustre: Skipped 1 previous similar message [ 4881.652079] Lustre: server umount lustre-MDT0001 complete [ 4883.932135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4886.802830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4900.319666] LustreError: 300437:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000cf7504d9 x1858577984674048/t0(0) o250->MGC192.168.204.158@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4900.427735] LustreError: 302730:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4900.434946] LustreError: 302730:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4900.453636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4900.479113] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4901.753024] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4904.459435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4904.579485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4905.636932] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4905.886212] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4907.613489] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4910.756092] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4912.217104] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 4913.428971] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4913.996374] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4915.679788] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4915.681972] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4915.685668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4915.687538] Lustre: Skipped 1 previous similar message [ 4921.367237] Lustre: server umount lustre-OST0000 complete [ 4925.408073] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4925.410763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4925.413806] Lustre: Skipped 1 previous similar message [ 4925.415283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4925.417530] Lustre: Skipped 4 previous similar messages [ 4928.727727] Lustre: server umount lustre-MDT0000 complete [ 4929.908402] LustreError: 301108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772478183 with bad export cookie 13168618014695486312 [ 4929.909750] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4929.911850] LustreError: 301108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4930.033419] Lustre: server umount lustre-MDT0001 complete [ 4933.499516] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 4934.566599] Key type lgssc unregistered [ 4934.697306] LNet: 306833:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4934.699548] LNetError: 306833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4934.706263] LNet: Removed LNI 192.168.204.158@tcp [ 4935.014089] Key type .llcrypt unregistered [ 4935.015218] Key type ._llcrypt unregistered [ 4944.214465] Key type ._llcrypt registered [ 4944.215507] Key type .llcrypt registered [ 4944.251728] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 4944.573079] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4944.611958] alg: No test for adler32 (adler32-zlib) [ 4945.468363] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 4945.545268] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 4947.127128] Key type lgssc registered [ 4947.483855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4949.979390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4949.984646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4951.134713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4952.389372] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4955.190668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4955.317498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4956.607568] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4958.356654] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4959.551956] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4960.735783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4960.738483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4960.742836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4965.855978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4965.856720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4966.677712] Lustre: server umount lustre-MDT0000 complete [ 4967.808237] LustreError: 309601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772478221 with bad export cookie 14209817602142343035 [ 4967.812129] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4967.813309] LustreError: 309601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4967.927480] Lustre: server umount lustre-MDT0001 complete [ 4972.477794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4972.573511] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4972.584922] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4972.660233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4973.855724] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4976.661401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4976.700199] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4976.714205] Lustre: 310566:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4976.815793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4978.074549] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4979.759387] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4980.941707] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4983.627849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4983.695149] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4983.698503] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4983.702384] LustreError: 312225:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -2 [ 4983.705637] LustreError: 312225:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 4983.708505] LustreError: 312225:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4983.732394] Lustre: server umount lustre-OST0000 complete [ 4983.734335] LustreError: 312225:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4986.423905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4986.488845] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4986.548061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4987.528491] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 4988.222091] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4989.865491] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4992.487084] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4992.522376] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4992.553907] Lustre: lustre-OST0001: new disk, initializing [ 4992.555633] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4994.434214] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4996.125101] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4997.116419] LustreError: 312689:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4997.120832] LustreError: 312689:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4997.126661] Lustre: server umount lustre-OST0001 complete [ 5002.210241] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5002.211723] LustreError: 313656:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5002.212121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5002.212128] Lustre: Skipped 1 previous similar message [ 5002.214588] Lustre: Skipped 1 previous similar message [ 5002.218330] LustreError: 313656:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5004.422951] Lustre: server umount lustre-OST0000 complete [ 5007.329791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5007.330793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5007.333417] Lustre: Skipped 1 previous similar message [ 5007.335058] Lustre: Skipped 1 previous similar message [ 5011.799425] Lustre: server umount lustre-MDT0000 complete [ 5012.448482] LustreError: 310580:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5012.452368] LustreError: 310580:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5013.012608] LustreError: 311292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772478266 with bad export cookie 14209817602142343574 [ 5013.017122] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5013.017966] LustreError: 311292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5013.132617] Lustre: server umount lustre-MDT0001 complete [ 5016.753947] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 5017.731587] Key type lgssc unregistered [ 5017.852363] LNet: 315471:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5017.854372] LNetError: 315471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5017.864316] LNet: Removed LNI 192.168.204.158@tcp [ 5018.154364] Key type .llcrypt unregistered [ 5018.155287] Key type ._llcrypt unregistered [ 5048.015213] Key type ._llcrypt registered [ 5048.016270] Key type .llcrypt registered [ 5048.050213] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 5054.283488] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 5054.825351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5054.837718] alg: No test for adler32 (adler32-zlib) [ 5055.683181] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 5055.766192] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 5057.343126] Key type lgssc registered [ 5057.679184] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5068.930116] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5068.935517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5070.037186] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5070.058187] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5070.106601] Lustre: lustre-MDT0000: new disk, initializing [ 5070.135543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5070.144113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5071.527400] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5075.796948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5075.838257] Lustre: 320338:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5075.863300] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5075.864990] Lustre: Skipped 1 previous similar message [ 5075.900129] Lustre: lustre-MDT0001: new disk, initializing [ 5075.932797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5075.943889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5075.947046] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5077.237183] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5080.204376] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5081.590716] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5083.787126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5083.918127] Lustre: lustre-OST0000: new disk, initializing [ 5083.919879] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5083.951878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5085.185778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5085.191199] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5085.232578] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5085.843735] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5088.925099] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5090.247969] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5090.318886] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5091.606513] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5091.674480] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5095.394203] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5095.394532] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5095.398262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5095.399533] LustreError: Skipped 1 previous similar message [ 5098.447513] Lustre: server umount lustre-OST0000 complete [ 5100.514468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5100.515863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5100.523694] Lustre: Skipped 2 previous similar messages [ 5100.527719] Lustre: Skipped 1 previous similar message [ 5105.632212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5105.633847] Lustre: Skipped 2 previous similar messages [ 5105.822183] Lustre: server umount lustre-MDT0000 complete [ 5107.035210] LustreError: 322078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772478360 with bad export cookie 2600958316943302877 [ 5107.039244] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5107.039904] LustreError: 322078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5107.158605] Lustre: server umount lustre-MDT0001 complete [ 5110.088804] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 14:06:02 (1772478362) [ 5112.426778] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 14:06:04 (1772478364) [ 5115.232942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5115.408283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5116.656065] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5119.321653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5119.439237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5120.697069] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5122.393716] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5123.575868] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5125.812239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5127.804894] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5129.603289] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5148.599537] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5149.171205] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5149.739628] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5150.336956] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5150.965299] Lustre: Failing over lustre-MDT0000 [ 5151.155415] Lustre: server umount lustre-MDT0000 complete [ 5155.296063] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5155.298918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5155.303432] LustreError: 324683:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5155.308497] LustreError: 324683:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5155.813214] LustreError: 324678:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5155.822380] LustreError: 324678:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5160.544614] LustreError: 324679:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5164.489663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5164.534808] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5164.652829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5164.656468] Lustre: Skipped 1 previous similar message [ 5164.686730] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5165.665266] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5166.037089] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5170.155183] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5170.174552] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5170.205873] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5171.528698] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5172.115578] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5174.062528] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5174.769964] Lustre: Failing over lustre-MDT0000 [ 5175.049584] Lustre: server umount lustre-MDT0000 complete [ 5175.265741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5175.266372] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5175.266548] LustreError: 326055:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5175.266558] LustreError: 326055:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5175.268888] Lustre: Skipped 4 previous similar messages [ 5180.385219] LustreError: 324682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5180.396431] LustreError: 324682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5187.830668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5187.872585] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5187.968318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5188.003459] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5189.179892] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5191.265406] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5193.188582] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5193.190241] Lustre: Skipped 2 previous similar messages [ 5193.198375] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5193.216792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5194.407753] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5194.966463] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5198.308909] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5198.309917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5198.311321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5198.312142] Lustre: Skipped 2 previous similar messages [ 5203.423300] Lustre: server umount lustre-MDT0000 complete [ 5203.424557] LustreError: 324682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5203.428885] LustreError: 324682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 5204.709430] LustreError: 326299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772478458 with bad export cookie 2600958316943305558 [ 5204.712240] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5204.713410] LustreError: 326299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5204.861294] Lustre: server umount lustre-MDT0001 complete [ 5216.752515] Lustre: server umount lustre-OST0000 complete [ 5223.094263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5223.246160] LustreError: 330478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5223.266833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5224.591027] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5228.250631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5229.698716] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5231.891116] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5233.570112] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5236.333479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5239.010188] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5241.122733] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5248.806623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5254.118077] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5254.119929] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5254.121030] LustreError: Skipped 1 previous similar message [ 5254.122587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5254.128816] Lustre: Skipped 1 previous similar message [ 5254.130510] Lustre: Skipped 2 previous similar messages [ 5259.222614] Lustre: server umount lustre-OST0000 complete [ 5264.352993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5264.353760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5264.357775] Lustre: Skipped 2 previous similar messages [ 5266.643207] Lustre: server umount lustre-MDT0000 complete [ 5268.105864] LustreError: 330460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772478521 with bad export cookie 2600958316943306279 [ 5268.110951] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5268.111836] LustreError: 330460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5268.245840] Lustre: server umount lustre-MDT0001 complete [ 5271.837222] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 5272.974699] Key type lgssc unregistered [ 5273.121467] LNet: 333676:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5273.123972] LNetError: 333676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5273.134451] LNet: Removed LNI 192.168.204.158@tcp [ 5273.496435] Key type .llcrypt unregistered [ 5273.498050] Key type ._llcrypt unregistered [ 5283.300847] Key type ._llcrypt registered [ 5283.301825] Key type .llcrypt registered [ 5283.336379] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 5283.688140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5283.707069] alg: No test for adler32 (adler32-zlib) [ 5284.566121] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 5284.645226] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 5286.223120] Key type lgssc registered [ 5286.723986] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5289.846561] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5289.858848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5291.046482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5292.255339] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5295.147333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5295.295973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5296.529346] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5298.273781] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5299.542579] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5301.858870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5301.973973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5303.891634] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5305.846257] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5315.310279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5320.673664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5320.674544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5320.678936] Lustre: Skipped 2 previous similar messages [ 5320.680755] Lustre: Skipped 1 previous similar message [ 5325.792287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5325.794262] Lustre: Skipped 3 previous similar messages [ 5326.104182] Lustre: server umount lustre-MDT0000 complete [ 5327.347827] LustreError: 335134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772478581 with bad export cookie 8534027321648465174 [ 5327.352107] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5327.355813] LustreError: 335134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5327.506633] Lustre: server umount lustre-MDT0001 complete [ 5339.421194] Lustre: server umount lustre-OST0000 complete [ 5343.559555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5343.738485] LustreError: 338547:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5343.744467] LustreError: 338547:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5343.763730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5345.214663] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5348.470092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5348.627851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5349.840778] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5351.881898] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5353.193617] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5356.055128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5358.467592] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5360.418583] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5368.555184] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5379.044408] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5379.045838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5379.050097] Lustre: Skipped 2 previous similar messages [ 5379.054854] Lustre: Skipped 2 previous similar messages [ 5383.827893] Lustre: server umount lustre-MDT0000 complete [ 5384.161539] LustreError: 338547:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5384.166142] LustreError: 338547:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5384.997123] LustreError: 338528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772478638 with bad export cookie 8534027321648466280 [ 5385.000081] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5385.002370] LustreError: 338528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5385.098568] Lustre: server umount lustre-MDT0001 complete [ 5396.498557] Lustre: server umount lustre-OST0000 complete [ 5401.166934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5401.309989] LustreError: 342137:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5401.333824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5401.336593] Lustre: Skipped 1 previous similar message [ 5402.491964] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5405.030481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5406.339120] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5407.906364] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5409.012742] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5411.095452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5412.261308] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5412.853885] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5414.446731] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5417.440247] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5417.443140] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5417.447744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5420.514578] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5420.516587] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5421.574805] Lustre: server umount lustre-OST0000 complete [ 5425.633081] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5425.633516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5425.636041] Lustre: Skipped 1 previous similar message [ 5425.636235] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5425.637523] Lustre: Skipped 3 previous similar messages [ 5428.832358] Lustre: server umount lustre-MDT0000 complete [ 5429.939115] LustreError: 343751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772478683 with bad export cookie 8534027321648467288 [ 5429.941736] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5429.941962] LustreError: 343751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5430.052168] Lustre: server umount lustre-MDT0001 complete [ 5433.121370] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 5433.958384] Key type lgssc unregistered [ 5434.068346] LNet: 345311:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5434.070275] LNetError: 345311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5434.080286] LNet: Removed LNI 192.168.204.158@tcp [ 5434.317516] Key type .llcrypt unregistered [ 5434.318468] Key type ._llcrypt unregistered [ 5442.629669] Key type ._llcrypt registered [ 5442.630567] Key type .llcrypt registered [ 5442.661848] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 5442.932532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5442.937776] alg: No test for adler32 (adler32-zlib) [ 5443.774941] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 5443.846721] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 5445.423110] Key type lgssc registered [ 5445.766662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5448.127178] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5448.132400] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5448.134608] LustreError: 346711:0:(osd_handler.c:8531:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5448.141860] LustreError: 346711:0:(obd_config.c:880:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5448.143637] LustreError: 346711:0:(obd_mount.c:217:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5448.145460] LustreError: 346711:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5448.147539] LustreError: 346711:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5450.347627] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5450.351311] LustreError: 347108:0:(osd_handler.c:8531:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5450.359764] LustreError: 347108:0:(obd_config.c:880:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5450.362550] LustreError: 347108:0:(obd_mount.c:217:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5450.365504] LustreError: 347108:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5450.368871] LustreError: 347108:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5455.917423] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 5456.751099] Key type lgssc unregistered [ 5456.881330] LNet: 348093:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5456.884477] LNetError: 348093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5456.898297] LNet: Removed LNI 192.168.204.158@tcp [ 5457.156351] Key type .llcrypt unregistered [ 5457.157269] Key type ._llcrypt unregistered [ 5462.393625] Key type ._llcrypt registered [ 5462.394804] Key type .llcrypt registered [ 5462.439482] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 5468.350767] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 5468.672282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5468.715402] alg: No test for adler32 (adler32-zlib) [ 5469.556370] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 5469.630344] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 5471.207108] Key type lgssc registered [ 5471.503319] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5481.671645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5481.675978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5482.760533] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5482.770462] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5482.805243] Lustre: lustre-MDT0000: new disk, initializing [ 5482.826677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5482.832478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5483.912128] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5487.622385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5487.658803] Lustre: 351188:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5487.678764] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5487.680216] Lustre: Skipped 1 previous similar message [ 5487.714962] Lustre: lustre-MDT0001: new disk, initializing [ 5487.744095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5487.754562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5487.757124] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5488.821433] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5491.546364] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5492.571167] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5494.547017] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5494.646315] Lustre: lustre-OST0000: new disk, initializing [ 5494.648100] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5494.677954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5496.358811] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5496.575722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5496.578761] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5496.621658] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5499.222392] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5500.414973] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5500.486651] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5501.681486] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5501.751451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5507.042477] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5507.042834] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5507.044487] LustreError: Skipped 1 previous similar message [ 5507.050026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5507.051816] Lustre: Skipped 1 previous similar message [ 5508.434896] Lustre: server umount lustre-OST0000 complete [ 5512.161453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5512.162687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5512.164401] Lustre: Skipped 2 previous similar messages [ 5515.613505] Lustre: server umount lustre-MDT0000 complete [ 5516.655497] LustreError: 351965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772478770 with bad export cookie 3057480615022426986 [ 5516.658203] LustreError: 351965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5516.659984] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5516.779538] Lustre: server umount lustre-MDT0001 complete [ 5518.941759] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5519.362483] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 14:12:51 (1772478771) [ 5521.258399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5521.436821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5522.640170] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5525.159978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5526.445641] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5527.918932] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5528.918559] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5530.872567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5530.979150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5530.982743] Lustre: Skipped 1 previous similar message [ 5532.702689] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5534.583223] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5537.256621] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5537.309913] Lustre: lustre-OST0001: new disk, initializing [ 5537.311504] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5538.986305] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5540.030436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5540.033228] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5540.063366] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5541.906906] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5545.442283] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5545.443081] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5545.444094] LustreError: Skipped 1 previous similar message [ 5545.447735] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5545.449563] Lustre: Skipped 2 previous similar messages [ 5548.135751] Lustre: lustre-OST0001: Not available for connect from 192.168.204.58@tcp (stopping) [ 5549.753807] Lustre: server umount lustre-OST0001 complete [ 5550.562334] LustreError: 356827:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5550.571067] LustreError: 356827:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5555.681849] LustreError: 356827:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5555.689238] LustreError: 356827:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5560.804417] LustreError: 358544:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5560.811611] LustreError: 358544:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5561.823747] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5561.825692] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5561.828449] Lustre: Skipped 1 previous similar message [ 5561.829841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5565.922201] LustreError: 358176:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5565.926009] LustreError: 358176:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5567.756832] Lustre: server umount lustre-OST0000 complete [ 5571.040945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5571.041475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5571.044098] Lustre: Skipped 2 previous similar messages [ 5571.045574] Lustre: Skipped 3 previous similar messages [ 5575.252840] Lustre: server umount lustre-MDT0000 complete [ 5576.159971] LustreError: 355238:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5576.163947] LustreError: 355238:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5576.351199] LustreError: 355954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772478830 with bad export cookie 3057480615022427854 [ 5576.354054] LustreError: 355954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5576.354444] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5582.505434] Lustre: server umount lustre-MDT0001 complete [ 5585.823655] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 5586.681514] Key type lgssc unregistered [ 5586.795273] LNet: 359587:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5586.797277] LNetError: 359587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5586.806280] LNet: Removed LNI 192.168.204.158@tcp [ 5587.050542] Key type .llcrypt unregistered [ 5587.051536] Key type ._llcrypt unregistered [ 5603.553990] Key type ._llcrypt registered [ 5603.554967] Key type .llcrypt registered [ 5603.586784] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 5603.929379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5603.951182] alg: No test for adler32 (adler32-zlib) [ 5604.786476] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 5604.858771] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 5606.431126] Key type lgssc registered [ 5606.723057] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5608.934893] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5608.940376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5610.028132] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5610.035302] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5610.092241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5611.214879] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5613.702877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5613.734306] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5613.743567] Lustre: 362343:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5613.833052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5614.996962] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5616.613538] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5617.732615] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5619.167836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5619.169744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5619.173141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5623.264232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5623.264687] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5624.527215] Lustre: server umount lustre-MDT0000 complete [ 5625.624043] LustreError: 362335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772478879 with bad export cookie 11118026551627475151 [ 5625.626846] LustreError: 362335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5625.626952] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5625.725382] Lustre: server umount lustre-MDT0001 complete [ 5628.524804] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 14:14:40 (1772478880) [ 5630.472363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5630.641362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5631.785272] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5634.203189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5634.317503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5635.427021] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5636.962043] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5638.047172] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5640.055682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5640.120458] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5640.174515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5641.780971] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5643.278676] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5651.811172] Lustre: Permanently deactivating lustre-OST0000 [ 5651.816289] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5661.677035] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5662.380516] LustreError: 367466:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5662.383523] LustreError: 367466:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5662.385707] LustreError: 367466:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5662.918374] LustreError: 367514:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5662.920274] LustreError: 367514:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5669.575091] Lustre: server umount lustre-OST0000 complete [ 5670.353200] LustreError: 367763:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5670.355209] LustreError: 367763:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5674.976336] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5674.977232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5674.979284] Lustre: Skipped 1 previous similar message [ 5674.981743] Lustre: Skipped 2 previous similar messages [ 5680.096044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5680.097648] Lustre: Skipped 1 previous similar message [ 5685.215076] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5685.216074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5685.218702] Lustre: Skipped 1 previous similar message [ 5685.272686] Lustre: server umount lustre-MDT0000 complete [ 5686.368092] LustreError: 364605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772478940 with bad export cookie 11118026551627475718 [ 5686.370806] LustreError: 364605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5686.370813] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5697.035289] Lustre: server umount lustre-MDT0001 complete [ 5699.642141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5700.785099] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5701.639180] LustreError: 368902:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5703.591611] Lustre: 369240:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.158@tcp [ 5704.220764] Lustre: server umount MGS complete [ 5707.618273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5707.761433] LustreError: 369964:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5707.765521] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5707.765927] LustreError: 369964:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5707.767382] Lustre: Skipped 1 previous similar message [ 5707.786460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5708.911582] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5711.433891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5711.525157] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5712.665787] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5714.256189] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5715.367027] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5715.943843] Lustre: Permanently reactivating lustre-OST0000 [ 5715.948366] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5715.950033] Lustre: Skipped 2 previous similar messages [ 5742.047106] LustreError: 371565:0:(osp_dev.c:694:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5744.802451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5744.921961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5744.923696] Lustre: Skipped 1 previous similar message [ 5746.707288] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5748.498802] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5761.505559] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5761.505795] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5761.509305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5761.510476] LustreError: Skipped 1 previous similar message [ 5761.513017] Lustre: Skipped 1 previous similar message [ 5762.380643] Lustre: server umount lustre-OST0000 complete [ 5765.589790] Lustre: server umount lustre-MDT0000 complete [ 5766.624367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5766.624941] LustreError: 369959:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5766.627214] Lustre: Skipped 2 previous similar messages [ 5766.697189] LustreError: 373292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772479020 with bad export cookie 11118026551627476754 [ 5766.699958] LustreError: 373292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5766.700906] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5766.801385] Lustre: server umount lustre-MDT0001 complete [ 5770.037030] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 5770.901209] Key type lgssc unregistered [ 5771.015310] LNet: 374326:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5771.017383] LNetError: 374326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5771.025336] LNet: Removed LNI 192.168.204.158@tcp [ 5771.266903] Key type .llcrypt unregistered [ 5771.267853] Key type ._llcrypt unregistered [ 5776.740344] Key type ._llcrypt registered [ 5776.741293] Key type .llcrypt registered [ 5776.775296] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 5782.504659] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 5782.920052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5782.929609] alg: No test for adler32 (adler32-zlib) [ 5783.779952] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 5783.861808] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 5785.439114] Key type lgssc registered [ 5785.816223] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5796.006713] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5796.011993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5797.118206] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5797.129286] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5797.168797] Lustre: lustre-MDT0000: new disk, initializing [ 5797.193647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5797.200106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5798.449628] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5802.265344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5802.305256] Lustre: 377421:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5802.326510] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5802.328159] Lustre: Skipped 1 previous similar message [ 5802.364368] Lustre: lustre-MDT0001: new disk, initializing [ 5802.398700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5802.409731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5802.414508] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5803.554837] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5806.440028] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5807.534822] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5809.898825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5809.999545] Lustre: lustre-OST0000: new disk, initializing [ 5810.001359] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5810.025463] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5811.688498] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5814.484702] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5815.639328] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5818.869186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5818.871951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5818.899400] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5819.762825] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 5820.868771] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5820.930301] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5823.969813] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5823.970356] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5823.971073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5823.971742] LustreError: Skipped 1 previous similar message [ 5823.974675] Lustre: Skipped 1 previous similar message [ 5827.600138] Lustre: server umount lustre-OST0000 complete [ 5829.089430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5829.090659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5829.092531] Lustre: Skipped 1 previous similar message [ 5829.096562] Lustre: Skipped 2 previous similar messages [ 5834.208107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5834.209753] Lustre: Skipped 1 previous similar message [ 5834.838301] Lustre: server umount lustre-MDT0000 complete [ 5835.904240] LustreError: 377412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772479089 with bad export cookie 5457987084467968382 [ 5835.907051] LustreError: 377412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5835.907202] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5836.013394] Lustre: server umount lustre-MDT0001 complete [ 5838.333427] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5838.790224] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 14:18:11 (1772479091) [ 5840.745213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5840.909676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5842.012565] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5844.531465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5845.860564] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5847.494751] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5848.516825] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5850.472877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5850.574298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5850.577409] Lustre: Skipped 1 previous similar message [ 5852.188235] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5853.683435] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5854.238871] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5855.641477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5859.809867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5859.810555] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5859.811453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5859.812830] Lustre: Skipped 1 previous similar message [ 5862.863441] Lustre: server umount lustre-OST0000 complete [ 5864.928650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5864.929454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5864.931977] Lustre: Skipped 1 previous similar message [ 5864.933402] Lustre: Skipped 1 previous similar message [ 5875.168090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5875.169657] Lustre: Skipped 4 previous similar messages [ 5878.239113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5878.290904] Lustre: server umount lustre-MDT0000 complete [ 5879.264352] LustreError: 381517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5879.268076] LustreError: 381517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5879.382975] LustreError: 382214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772479133 with bad export cookie 5457987084467969250 [ 5879.385733] LustreError: 382214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5879.386710] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5879.489796] Lustre: server umount lustre-MDT0001 complete [ 5882.475052] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 5883.422428] Key type lgssc unregistered [ 5883.561365] LNet: 384869:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5883.563597] LNetError: 384869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5883.572369] LNet: Removed LNI 192.168.204.158@tcp [ 5883.849025] Key type .llcrypt unregistered [ 5883.849944] Key type ._llcrypt unregistered [ 5890.666632] Key type ._llcrypt registered [ 5890.667625] Key type .llcrypt registered [ 5890.708129] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 5890.805328] Key type .llcrypt unregistered [ 5890.806595] Key type ._llcrypt unregistered [ 5897.116014] Key type ._llcrypt registered [ 5897.117121] Key type .llcrypt registered [ 5897.153170] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 5897.710642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5897.724082] alg: No test for adler32 (adler32-zlib) [ 5898.570375] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 5898.656292] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 5900.231197] Key type lgssc registered [ 5900.783557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5903.578898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5903.584810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5904.738441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5905.867540] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5908.277697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5908.382923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5909.969920] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5911.477214] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5913.423134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5913.539532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5914.599213] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5914.676905] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 5918.688254] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5918.690905] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5918.694320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5919.714263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5919.715155] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5919.715155] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5919.717644] Lustre: Skipped 1 previous similar message [ 5923.402891] Lustre: server umount lustre-OST0000 complete [ 5924.832287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5924.832949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5924.835531] Lustre: Skipped 1 previous similar message [ 5929.951976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5929.953573] Lustre: Skipped 2 previous similar messages [ 5930.642405] Lustre: server umount lustre-MDT0000 complete [ 5931.728266] LustreError: 387002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772479185 with bad export cookie 13822780629828394626 [ 5931.731603] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5931.732806] LustreError: 387002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5931.840783] Lustre: server umount lustre-MDT0001 complete [ 5934.838364] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 5935.656613] Key type lgssc unregistered [ 5935.772400] LNet: 390009:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5935.774295] LNetError: 390009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5935.781272] LNet: Removed LNI 192.168.204.158@tcp [ 5936.037079] Key type .llcrypt unregistered [ 5936.038028] Key type ._llcrypt unregistered [ 5943.806364] Key type ._llcrypt registered [ 5943.807454] Key type .llcrypt registered [ 5943.839592] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 5944.139219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5944.182991] alg: No test for adler32 (adler32-zlib) [ 5945.017432] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 5945.088181] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 5946.663104] Key type lgssc registered [ 5946.956437] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5949.123589] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5949.128815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5964.575248] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5969.695125] LustreError: 391360:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.158@tcp: failed processing log, type 4: rc = -110 [ 5996.319266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5998.193565] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6000.235297] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6002.859701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6003.055303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6003.088100] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6004.256241] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6008.169691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6008.288866] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6008.395699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6010.109446] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6012.798850] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6014.640361] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6018.529187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6018.529663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6018.532951] LustreError: Skipped 1 previous similar message [ 6018.533669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6018.540330] Lustre: Skipped 1 previous similar message [ 6018.548516] Lustre: Skipped 1 previous similar message [ 6021.091584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6021.096990] Lustre: Skipped 1 previous similar message [ 6023.663260] Lustre: server umount lustre-OST0000 complete [ 6026.210603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6026.211789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6026.218063] Lustre: Skipped 1 previous similar message [ 6026.224257] Lustre: Skipped 1 previous similar message [ 6031.329303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6031.333753] Lustre: Skipped 1 previous similar message [ 6031.662291] Lustre: server umount lustre-MDT0000 complete [ 6033.457035] LustreError: 391367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772479287 with bad export cookie 1205525521684023232 [ 6033.461148] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6033.463278] LustreError: 391367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6033.605973] Lustre: server umount lustre-MDT0001 complete [ 6037.999815] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 6039.299542] Key type lgssc unregistered [ 6039.476602] LNet: 394576:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6039.482105] LNetError: 394576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6039.495575] LNet: Removed LNI 192.168.204.158@tcp [ 6039.858192] Key type .llcrypt unregistered [ 6039.859127] Key type ._llcrypt unregistered [ 6051.057735] Key type ._llcrypt registered [ 6051.059115] Key type .llcrypt registered [ 6051.093474] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 6051.479078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6051.484899] alg: No test for adler32 (adler32-zlib) [ 6052.353492] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 6052.447367] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 6054.039158] Key type lgssc registered [ 6054.443723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6057.990460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6058.001141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6059.180601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6060.827404] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6065.083393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6065.284535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6066.974799] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6069.571771] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6071.343117] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6074.364632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6074.467838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6076.356621] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6078.286321] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6084.717372] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6090.209952] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6090.213271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6090.217422] Lustre: Skipped 2 previous similar messages [ 6090.221043] Lustre: Skipped 1 previous similar message [ 6093.936807] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.58@tcp (stopping) [ 6095.330651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6095.335460] Lustre: Skipped 2 previous similar messages [ 6096.249068] Lustre: server umount lustre-MDT0000 complete [ 6097.353523] LustreError: 395934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.58@tcp arrived at 1772479351 with bad export cookie 1215063443291701257 [ 6097.365035] LustreError: 395934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6098.307901] LustreError: 395932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772479352 with bad export cookie 1215063443291700634 [ 6098.314747] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6098.314996] LustreError: 395932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6100.448339] LustreError: 395951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6100.449169] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6100.455138] LustreError: 395951:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6100.464262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6103.456396] LustreError: 395952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6103.465781] LustreError: 395952:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6104.423437] Lustre: server umount lustre-OST0000 complete [ 6107.698205] Lustre: server umount lustre-MDT0001 complete [ 6112.502457] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 6113.879306] Key type lgssc unregistered [ 6114.011705] LNet: 399302:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6114.016844] LNetError: 399302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6114.028555] LNet: Removed LNI 192.168.204.158@tcp [ 6114.471777] Key type .llcrypt unregistered [ 6114.473924] Key type ._llcrypt unregistered [ 6126.686174] Key type ._llcrypt registered [ 6126.687938] Key type .llcrypt registered [ 6126.749834] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 6127.332966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6127.348613] alg: No test for adler32 (adler32-zlib) [ 6128.203866] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 6128.296513] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 6129.871137] Key type lgssc registered [ 6130.527291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6134.351354] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6134.362115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6135.567902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6136.972438] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6141.029525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6141.188291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6143.024103] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6145.696568] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6147.524971] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6150.585093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6150.738216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6153.349838] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6155.926652] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6160.998477] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6165.983973] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6165.986549] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6165.990620] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6166.498831] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6166.501373] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6168.428266] Lustre: server umount lustre-MDT0001 complete [ 6171.617724] LustreError: 401790:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6171.618363] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6171.620223] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6171.621165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6171.621172] Lustre: Skipped 2 previous similar messages [ 6171.627495] LustreError: 401790:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6176.668089] Lustre: server umount lustre-OST0000 complete [ 6176.737030] LustreError: 400678:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6178.345197] Lustre: server umount lustre-MDT0000 complete [ 6184.046573] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 6185.171860] Key type lgssc unregistered [ 6185.327790] LNet: 404011:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6185.333448] LNetError: 404011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6185.344616] LNet: Removed LNI 192.168.204.158@tcp [ 6185.804374] Key type .llcrypt unregistered [ 6185.806851] Key type ._llcrypt unregistered [ 6196.678544] Key type ._llcrypt registered [ 6196.680476] Key type .llcrypt registered [ 6196.729488] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 6204.414450] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 6204.849392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6204.864057] alg: No test for adler32 (adler32-zlib) [ 6205.735322] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 6205.824226] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 6207.399144] Key type lgssc registered [ 6207.939811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6224.537627] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 6229.207536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6229.212794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6230.302767] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6230.315840] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6230.355927] Lustre: lustre-MDT0000: new disk, initializing [ 6230.383157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6230.391166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6231.635351] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6236.903367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6237.011894] Lustre: lustre-OST0000: new disk, initializing [ 6237.013764] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6237.015433] Lustre: Skipped 1 previous similar message [ 6237.040058] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6238.769018] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6241.750427] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6243.840753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6243.843355] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6243.851333] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6252.056595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6252.100463] Lustre: 407945:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6252.134234] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6252.169397] Lustre: lustre-MDT0001: new disk, initializing [ 6252.219546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6252.234684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6252.237900] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6253.436495] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6256.981349] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6257.076855] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6258.371455] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6258.489506] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6262.424454] Lustre: server umount lustre-MDT0001 complete [ 6262.755564] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6262.757510] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6262.760989] Lustre: Skipped 1 previous similar message [ 6267.876043] LustreError: 410936:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6267.882790] LustreError: 410936:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6272.996548] LustreError: 407952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6273.005394] LustreError: 407952:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6278.115679] LustreError: 407957:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6278.116432] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6278.138671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6283.233559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6283.234824] LustreError: 410248:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6288.353915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6288.355049] LustreError: 407957:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6289.375149] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6289.455955] Lustre: server umount lustre-OST0000 complete [ 6297.407921] Lustre: server umount lustre-MDT0000 complete [ 6302.664957] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 6304.002173] Key type lgssc unregistered [ 6304.191825] LNet: 412516:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6304.197448] LNetError: 412516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6304.207840] LNet: Removed LNI 192.168.204.158@tcp [ 6304.699517] Key type .llcrypt unregistered [ 6304.701919] Key type ._llcrypt unregistered [ 6318.804452] Key type ._llcrypt registered [ 6318.807100] Key type .llcrypt registered [ 6318.880868] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 6319.070353] Key type .llcrypt unregistered [ 6319.072625] Key type ._llcrypt unregistered [ 6326.135721] Key type ._llcrypt registered [ 6326.137583] Key type .llcrypt registered [ 6326.199116] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 6326.561616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6326.603801] alg: No test for adler32 (adler32-zlib) [ 6327.450545] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 6327.527983] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 6329.111166] Key type lgssc registered [ 6329.705988] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6341.434852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6341.441203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6342.553811] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6342.565204] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6342.601701] Lustre: lustre-MDT0000: new disk, initializing [ 6342.629325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6342.635686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6343.968920] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6348.852065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6348.890804] Lustre: 415833:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6348.912089] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6348.914361] Lustre: Skipped 1 previous similar message [ 6348.954058] Lustre: lustre-MDT0001: new disk, initializing [ 6348.983373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6348.993284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6348.997726] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6350.508332] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6353.746719] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6355.390781] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6358.239371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6358.337415] Lustre: lustre-OST0000: new disk, initializing [ 6358.339061] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6358.364778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6360.334285] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6363.708417] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6366.730365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6366.736838] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6366.790278] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6371.810587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6371.811833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6371.813889] Lustre: Skipped 2 previous similar messages [ 6371.816788] Lustre: Skipped 2 previous similar messages [ 6376.930324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6376.933938] Lustre: Skipped 2 previous similar messages [ 6377.714826] Lustre: server umount lustre-MDT0000 complete [ 6379.454858] LustreError: 415824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772479633 with bad export cookie 16113256811525896338 [ 6379.459371] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6379.461938] LustreError: 415824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6379.627298] Lustre: server umount lustre-MDT0001 complete [ 6391.745580] Lustre: server umount lustre-OST0000 complete [ 6396.924055] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 14:27:29 (1772479649) [ 6400.561850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6400.829456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6402.266694] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6406.112969] LustreError: 419492:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6406.122434] LustreError: 419492:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6406.236635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6406.424566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6408.175845] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6410.567603] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6412.370757] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6426.617907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6426.721798] Lustre: Found index 0 for lustre-OST0000, updating log [ 6426.805936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6429.310991] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 6431.855616] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6445.228464] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6450.660455] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6450.660651] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6450.668856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6450.677692] Lustre: Skipped 1 previous similar message [ 6451.497841] Lustre: server umount lustre-OST0000 complete [ 6466.784637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6466.785395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6466.787523] Lustre: Skipped 2 previous similar messages [ 6466.789949] Lustre: Skipped 1 previous similar message [ 6473.184980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6473.189171] Lustre: Skipped 3 previous similar messages [ 6479.327138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6479.400785] Lustre: server umount lustre-MDT0000 complete [ 6481.228735] LustreError: 419471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772479735 with bad export cookie 16113256811525897717 [ 6481.234554] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6481.237337] LustreError: 419471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6481.371189] Lustre: server umount lustre-MDT0001 complete [ 6484.939314] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 14:28:57 (1772479737) [ 6488.287195] LNet: 1 local NIs in recovery (showing 1): 192.168.204.158@tcp [ 7067.483055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7067.619590] LustreError: 425580:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7067.640773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7068.959961] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7072.156988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7072.288249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7073.712518] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7075.792442] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7077.035523] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7090.516582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7090.643546] Lustre: Found index 0 for lustre-OST0000, updating log [ 7090.762954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7092.911176] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7095.085123] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7095.900713] Lustre: server umount lustre-OST0000 complete [ 7097.824454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7097.827116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7097.831710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7097.834675] Lustre: Skipped 2 previous similar messages [ 7098.848427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7098.849369] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7103.380421] Lustre: server umount lustre-MDT0000 complete [ 7104.820270] LustreError: 426298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772480358 with bad export cookie 16113256811525898788 [ 7104.822907] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7104.824722] LustreError: 426298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7104.953813] Lustre: server umount lustre-MDT0001 complete [ 7123.141204] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 7126.577480] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 7128.287225] LNet: 1 local NIs in recovery (showing 1): 192.168.204.158@tcp [ 7139.602132] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 14:39:51 (1772480391) [ 7145.576240] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 7146.642047] Key type lgssc unregistered [ 7152.927598] LNet: 431830:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7152.930406] LNetError: 431830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7222.565380] LNet: Removed LNI 192.168.204.158@tcp [ 7222.827442] Key type .llcrypt unregistered [ 7222.828352] Key type ._llcrypt unregistered [ 7229.019121] Key type ._llcrypt registered [ 7229.019935] Key type .llcrypt registered [ 7229.055651] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 7229.480923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7229.487353] alg: No test for adler32 (adler32-zlib) [ 7230.345061] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 7230.427390] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 7232.007220] Key type lgssc registered [ 7232.382536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7233.865280] LNet: Added LNI 192.168.204.158@tcp100 [8/256/0/180] [ 7234.184047] LNet: Added LNI 192.168.204.158@tcp101 [8/256/0/180] [ 7234.506370] LNet: Added LNI 192.168.204.158@tcp102 [8/256/0/180] [ 7234.823392] LNet: Added LNI 192.168.204.158@tcp103 [8/256/0/180] [ 7235.154647] LNet: Added LNI 192.168.204.158@tcp104 [8/256/0/180] [ 7235.473234] LNet: Added LNI 192.168.204.158@tcp105 [8/256/0/180] [ 7235.811977] LNet: Added LNI 192.168.204.158@tcp106 [8/256/0/180] [ 7236.151071] LNet: Added LNI 192.168.204.158@tcp107 [8/256/0/180] [ 7236.468279] LNet: Added LNI 192.168.204.158@tcp108 [8/256/0/180] [ 7236.775422] LNet: Added LNI 192.168.204.158@tcp109 [8/256/0/180] [ 7237.097602] LNet: Added LNI 192.168.204.158@tcp110 [8/256/0/180] [ 7237.421833] LNet: Added LNI 192.168.204.158@tcp111 [8/256/0/180] [ 7237.725937] LNet: Added LNI 192.168.204.158@tcp112 [8/256/0/180] [ 7238.042035] LNet: Added LNI 192.168.204.158@tcp113 [8/256/0/180] [ 7238.347108] LNet: Added LNI 192.168.204.158@tcp114 [8/256/0/180] [ 7238.651620] LNet: Added LNI 192.168.204.158@tcp115 [8/256/0/180] [ 7238.941624] LNet: Added LNI 192.168.204.158@tcp116 [8/256/0/180] [ 7239.239131] LNet: Added LNI 192.168.204.158@tcp117 [8/256/0/180] [ 7239.544344] LNet: Added LNI 192.168.204.158@tcp118 [8/256/0/180] [ 7239.885869] LNet: Added LNI 192.168.204.158@tcp119 [8/256/0/180] [ 7240.182265] LNet: Added LNI 192.168.204.158@tcp120 [8/256/0/180] [ 7240.486110] LNet: Added LNI 192.168.204.158@tcp121 [8/256/0/180] [ 7240.781558] LNet: Added LNI 192.168.204.158@tcp122 [8/256/0/180] [ 7241.070637] LNet: Added LNI 192.168.204.158@tcp123 [8/256/0/180] [ 7241.364908] LNet: Added LNI 192.168.204.158@tcp124 [8/256/0/180] [ 7241.648392] LNet: Added LNI 192.168.204.158@tcp125 [8/256/0/180] [ 7241.931650] LNet: Added LNI 192.168.204.158@tcp126 [8/256/0/180] [ 7242.248488] LNet: Added LNI 192.168.204.158@tcp127 [8/256/0/180] [ 7242.555463] LNet: Added LNI 192.168.204.158@tcp128 [8/256/0/180] [ 7242.852624] LNet: Added LNI 192.168.204.158@tcp129 [8/256/0/180] [ 7243.144280] LNet: Added LNI 192.168.204.158@tcp130 [8/256/0/180] [ 7243.436258] LNet: Added LNI 192.168.204.158@tcp131 [8/256/0/180] [ 7243.728063] LNet: Added LNI 192.168.204.158@tcp132 [8/256/0/180] [ 7244.025519] LNet: Added LNI 192.168.204.158@tcp133 [8/256/0/180] [ 7244.321519] LNet: Added LNI 192.168.204.158@tcp134 [8/256/0/180] [ 7244.609429] LNet: Added LNI 192.168.204.158@tcp135 [8/256/0/180] [ 7246.819881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7246.824691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7247.870765] Lustre: 434812:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.158@tcp: can't add NID 192.168.204.158@tcp130: rc = -75 [ 7247.938325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7247.950280] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7247.989226] Lustre: lustre-MDT0000: new disk, initializing [ 7248.018455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7248.024916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7248.929624] LNetError: 432276:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.158 [ 7248.933219] LNetError: Protocol error connecting to 192.168.204.158@tcp113 at host 192.168.204.158:988: is it running a compatible version of Lustre? [ 7249.363215] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7253.634899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7253.685944] Lustre: 434830:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7253.709605] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7253.712180] Lustre: Skipped 1 previous similar message [ 7253.751984] Lustre: lustre-MDT0001: new disk, initializing [ 7253.765686] Lustre: 435612:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.158@tcp: can't add NID 192.168.204.158@tcp130: rc = -75 [ 7253.769328] Lustre: 435612:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7253.809544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7253.823322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7253.827850] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7255.153849] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7258.231670] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7259.402049] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7261.550762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7261.649926] Lustre: lustre-OST0000: new disk, initializing [ 7261.651813] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7261.687362] Lustre: 436574:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.158@tcp: can't add NID 192.168.204.158@tcp130: rc = -75 [ 7261.691239] Lustre: 436574:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7261.710721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7261.777644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7261.782992] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7261.852585] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7263.512168] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7266.743595] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7272.418803] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7272.418850] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7272.423915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7272.426044] LustreError: Skipped 1 previous similar message [ 7272.429681] Lustre: Skipped 1 previous similar message [ 7275.287759] Lustre: server umount lustre-OST0000 complete [ 7277.536547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7277.537219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7277.540722] Lustre: Skipped 2 previous similar messages [ 7278.560614] LNetError: 432275:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.158 [ 7278.564128] LNetError: Protocol error connecting to 192.168.204.158@tcp113 at host 192.168.204.158:988: is it running a compatible version of Lustre? [ 7282.656726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7282.659581] Lustre: Skipped 2 previous similar messages [ 7282.720544] Lustre: server umount lustre-MDT0000 complete [ 7283.963211] LustreError: 434822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772480537 with bad export cookie 291451597791815185 [ 7283.966377] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7283.967178] LustreError: 434822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7284.072478] Lustre: server umount lustre-MDT0001 complete [ 7287.578168] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 7288.596155] Key type lgssc unregistered [ 7288.765351] LNet: 438350:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7288.768821] LNetError: 438350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7288.770832] LNet: Removed LNI 192.168.204.158@tcp [ 7289.823170] LNet: Removed LNI 192.168.204.158@tcp100 [ 7289.824515] LNetError: 438350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7289.826363] LNetError: 438350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7289.828936] LNet: Removed LNI 192.168.204.158@tcp101 [ 7290.847133] LNet: Removed LNI 192.168.204.158@tcp102 [ 7290.848565] LNetError: 438350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7290.851974] LNetError: 438350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7290.855460] LNet: Removed LNI 192.168.204.158@tcp103 [ 7291.871133] LNet: Removed LNI 192.168.204.158@tcp104 [ 7292.895102] LNet: Removed LNI 192.168.204.158@tcp105 [ 7292.896662] LNetError: 438350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7292.899344] LNetError: 438350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7292.902296] LNet: Removed LNI 192.168.204.158@tcp106 [ 7293.919160] LNet: Removed LNI 192.168.204.158@tcp107 [ 7294.943097] LNet: Removed LNI 192.168.204.158@tcp108 [ 7295.969260] LNet: Removed LNI 192.168.204.158@tcp109 [ 7296.991090] LNet: Removed LNI 192.168.204.158@tcp110 [ 7296.992281] LNetError: 438350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7296.994449] LNetError: 438350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7296.997146] LNet: Removed LNI 192.168.204.158@tcp111 [ 7298.015134] LNet: Removed LNI 192.168.204.158@tcp112 [ 7299.039162] LNet: Removed LNI 192.168.204.158@tcp113 [ 7300.063101] LNet: Removed LNI 192.168.204.158@tcp114 [ 7301.087130] LNet: Removed LNI 192.168.204.158@tcp115 [ 7302.111160] LNet: Removed LNI 192.168.204.158@tcp116 [ 7302.113622] LNet: Removed LNI 192.168.204.158@tcp117 [ 7302.114543] LNet: Removed LNI 192.168.204.158@tcp118 [ 7302.115717] LNet: Removed LNI 192.168.204.158@tcp119 [ 7302.116817] LNet: Removed LNI 192.168.204.158@tcp120 [ 7302.118217] LNet: Removed LNI 192.168.204.158@tcp121 [ 7302.119418] LNet: Removed LNI 192.168.204.158@tcp122 [ 7302.121218] LNet: Removed LNI 192.168.204.158@tcp123 [ 7302.123038] LNet: Removed LNI 192.168.204.158@tcp124 [ 7302.124886] LNet: Removed LNI 192.168.204.158@tcp125 [ 7302.126233] LNet: Removed LNI 192.168.204.158@tcp126 [ 7302.127577] LNet: Removed LNI 192.168.204.158@tcp127 [ 7302.129078] LNet: Removed LNI 192.168.204.158@tcp128 [ 7302.130652] LNet: Removed LNI 192.168.204.158@tcp129 [ 7302.132151] LNet: Removed LNI 192.168.204.158@tcp130 [ 7302.133443] LNet: Removed LNI 192.168.204.158@tcp131 [ 7302.135128] LNet: Removed LNI 192.168.204.158@tcp132 [ 7302.136850] LNet: Removed LNI 192.168.204.158@tcp133 [ 7302.137970] LNet: Removed LNI 192.168.204.158@tcp134 [ 7302.145422] LNet: Removed LNI 192.168.204.158@tcp135 [ 7302.410495] Key type .llcrypt unregistered [ 7302.411863] Key type ._llcrypt unregistered [ 7309.834614] Key type ._llcrypt registered [ 7309.836113] Key type .llcrypt registered [ 7309.871939] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 7315.724863] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 7316.114616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7316.138520] alg: No test for adler32 (adler32-zlib) [ 7316.983866] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 7317.057753] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 7318.639143] Key type lgssc registered [ 7318.965774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7330.192215] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 14:43:02 (1772480582) [ 7336.262494] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 7337.237972] Key type lgssc unregistered [ 7337.358422] LNet: 442354:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7337.360874] LNetError: 442354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7337.370333] LNet: Removed LNI 192.168.204.158@tcp [ 7337.650377] Key type .llcrypt unregistered [ 7337.651737] Key type ._llcrypt unregistered [ 7344.210529] Key type ._llcrypt registered [ 7344.211398] Key type .llcrypt registered [ 7344.247856] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 7344.575697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7344.621571] alg: No test for adler32 (adler32-zlib) [ 7345.463060] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 7345.541827] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 7347.119175] Key type lgssc registered [ 7347.487320] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7350.625201] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7350.630904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7351.726605] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7351.738172] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7351.776332] Lustre: lustre-MDT0000: new disk, initializing [ 7351.804738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7351.812631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7353.036937] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7357.110718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7357.153600] Lustre: 443526:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7357.173508] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7357.175202] Lustre: Skipped 1 previous similar message [ 7357.209051] Lustre: lustre-MDT0001: new disk, initializing [ 7357.240913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7357.255945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7357.260439] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7358.544029] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7361.722879] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7362.999308] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7365.344753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7365.445166] Lustre: lustre-OST0000: new disk, initializing [ 7365.447646] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7365.472480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7367.346524] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7368.373912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7368.377323] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7368.405110] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7370.400739] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7371.591227] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing lnet_if_list [ 7373.208830] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7373.222570] LNet: Added LNI 192.168.204.158@tcp10 [8/256/0/180] [ 7374.927735] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7374.938441] LNet: Added LNI 192.168.204.158@tcp11 [8/256/0/180] [ 7376.269126] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7376.282948] LNet: Added LNI 192.168.204.158@tcp12 [8/256/0/180] [ 7377.650670] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7377.661145] LNet: Added LNI 192.168.204.158@tcp13 [8/256/0/180] [ 7378.980414] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7378.991328] LNet: Added LNI 192.168.204.158@tcp14 [8/256/0/180] [ 7380.318775] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7380.330260] LNet: Added LNI 192.168.204.158@tcp15 [8/256/0/180] [ 7381.693545] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7381.703413] LNet: Added LNI 192.168.204.158@tcp16 [8/256/0/180] [ 7383.063887] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7383.073149] LNet: Added LNI 192.168.204.158@tcp17 [8/256/0/180] [ 7384.461763] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7384.474165] LNet: Added LNI 192.168.204.158@tcp18 [8/256/0/180] [ 7385.894835] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7385.903912] LNet: Added LNI 192.168.204.158@tcp19 [8/256/0/180] [ 7387.309399] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7387.320116] LNet: Added LNI 192.168.204.158@tcp20 [8/256/0/180] [ 7388.695312] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7388.705597] LNet: Added LNI 192.168.204.158@tcp21 [8/256/0/180] [ 7390.177187] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7390.188501] LNet: Added LNI 192.168.204.158@tcp22 [8/256/0/180] [ 7391.801287] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7391.816458] LNet: Added LNI 192.168.204.158@tcp23 [8/256/0/180] [ 7393.405418] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7393.414314] LNet: Added LNI 192.168.204.158@tcp24 [8/256/0/180] [ 7395.037690] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7395.047370] LNet: Added LNI 192.168.204.158@tcp25 [8/256/0/180] [ 7396.729164] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7396.739815] LNet: Added LNI 192.168.204.158@tcp26 [8/256/0/180] [ 7398.489344] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7398.502315] LNet: Added LNI 192.168.204.158@tcp27 [8/256/0/180] [ 7400.245847] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7400.256552] LNet: Added LNI 192.168.204.158@tcp28 [8/256/0/180] [ 7401.986878] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7401.996827] LNet: Added LNI 192.168.204.158@tcp29 [8/256/0/180] [ 7403.720798] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7403.731997] LNet: Added LNI 192.168.204.158@tcp30 [8/256/0/180] [ 7405.546904] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7405.559171] LNet: Added LNI 192.168.204.158@tcp31 [8/256/0/180] [ 7407.474870] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7407.484753] LNet: Added LNI 192.168.204.158@tcp32 [8/256/0/180] [ 7409.413253] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7409.424389] LNet: Added LNI 192.168.204.158@tcp33 [8/256/0/180] [ 7411.428259] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7411.437474] LNet: Added LNI 192.168.204.158@tcp34 [8/256/0/180] [ 7413.331029] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7413.344199] LNet: Added LNI 192.168.204.158@tcp35 [8/256/0/180] [ 7415.320205] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7415.332308] LNet: Added LNI 192.168.204.158@tcp36 [8/256/0/180] [ 7417.374081] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7417.385794] LNet: Added LNI 192.168.204.158@tcp37 [8/256/0/180] [ 7419.458933] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7419.470177] LNet: Added LNI 192.168.204.158@tcp38 [8/256/0/180] [ 7421.635383] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7421.646817] LNet: Added LNI 192.168.204.158@tcp39 [8/256/0/180] [ 7423.817037] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7423.827472] LNet: Added LNI 192.168.204.158@tcp40 [8/256/0/180] [ 7426.085393] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7426.097879] LNet: Added LNI 192.168.204.158@tcp41 [8/256/0/180] [ 7428.490498] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7428.505816] LNet: Added LNI 192.168.204.158@tcp42 [8/256/0/180] [ 7430.864081] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7430.874115] LNet: Added LNI 192.168.204.158@tcp43 [8/256/0/180] [ 7433.177096] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7433.187837] LNet: Added LNI 192.168.204.158@tcp44 [8/256/0/180] [ 7435.547149] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7435.559212] LNet: Added LNI 192.168.204.158@tcp45 [8/256/0/180] [ 7437.928891] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7437.940396] LNet: Added LNI 192.168.204.158@tcp46 [8/256/0/180] [ 7440.333693] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7440.344503] LNet: Added LNI 192.168.204.158@tcp47 [8/256/0/180] [ 7442.810349] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7442.822504] LNet: Added LNI 192.168.204.158@tcp48 [8/256/0/180] [ 7445.407615] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7445.418019] LNet: Added LNI 192.168.204.158@tcp49 [8/256/0/180] [ 7447.867546] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7447.877412] LNet: Added LNI 192.168.204.158@tcp50 [8/256/0/180] [ 7455.712495] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7455.714935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7455.717369] Lustre: Skipped 2 previous similar messages [ 7455.719712] Lustre: Skipped 1 previous similar message [ 7460.832574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7460.835209] Lustre: Skipped 3 previous similar messages [ 7461.277235] Lustre: server umount lustre-MDT0000 complete [ 7462.497083] LustreError: 444304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772480716 with bad export cookie 16424135726816398191 [ 7462.500572] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7462.501341] LustreError: 444304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7462.598750] Lustre: server umount lustre-MDT0001 complete [ 7469.605162] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 7472.523610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7473.119292] LustreError: 442798:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000bc815d85 x1858580616655872/t0(0) o250->MGC192.168.204.158@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7473.232554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7474.481372] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7477.216638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7477.305832] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7477.338151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7478.506979] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7480.123586] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7481.209464] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7481.311166] Lustre: 442802:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772480719/real 1772480719] req@00000000359ac83e x1858580616655488/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772480735 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7482.342239] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7483.359125] Lustre: 442799:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772480721/real 1772480721] req@00000000949968ee x1858580616655616/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772480737 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7485.412174] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7486.415106] Lustre: 442800:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772480724/real 1772480724] req@00000000abeff319 x1858580616655744/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772480740 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7488.479073] Lustre: 442801:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772480726/real 1772480726] req@0000000073128f09 x1858580616656000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772480742 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7490.529616] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7490.530891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7490.535586] Lustre: Skipped 1 previous similar message [ 7494.743399] Lustre: server umount lustre-OST0000 complete [ 7497.695809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7497.698123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7497.703335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7497.706596] Lustre: Skipped 1 previous similar message [ 7502.166754] Lustre: server umount lustre-MDT0000 complete [ 7503.441273] LustreError: 444304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772480757 with bad export cookie 16424135726816400221 [ 7503.444655] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7503.447749] LustreError: 444304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7503.556264] Lustre: server umount lustre-MDT0001 complete [ 7507.202475] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 7508.254504] Key type lgssc unregistered [ 7508.428079] LNet: 471185:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7508.431478] LNetError: 471185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7508.433984] LNet: Removed LNI 192.168.204.158@tcp [ 7508.435132] LNet: Removed LNI 192.168.204.158@tcp10 [ 7508.436438] LNet: Removed LNI 192.168.204.158@tcp11 [ 7508.437957] LNet: Removed LNI 192.168.204.158@tcp12 [ 7508.439710] LNet: Removed LNI 192.168.204.158@tcp13 [ 7508.441620] LNet: Removed LNI 192.168.204.158@tcp14 [ 7508.443196] LNet: Removed LNI 192.168.204.158@tcp15 [ 7508.444788] LNet: Removed LNI 192.168.204.158@tcp16 [ 7508.446130] LNet: Removed LNI 192.168.204.158@tcp17 [ 7508.447738] LNet: Removed LNI 192.168.204.158@tcp18 [ 7508.449204] LNet: Removed LNI 192.168.204.158@tcp19 [ 7508.451172] LNet: Removed LNI 192.168.204.158@tcp20 [ 7508.453355] LNet: Removed LNI 192.168.204.158@tcp21 [ 7508.455549] LNet: Removed LNI 192.168.204.158@tcp22 [ 7508.456770] LNet: Removed LNI 192.168.204.158@tcp23 [ 7508.458260] LNet: Removed LNI 192.168.204.158@tcp24 [ 7508.459718] LNet: Removed LNI 192.168.204.158@tcp25 [ 7508.461288] LNet: Removed LNI 192.168.204.158@tcp26 [ 7508.462829] LNet: Removed LNI 192.168.204.158@tcp27 [ 7508.464353] LNet: Removed LNI 192.168.204.158@tcp28 [ 7508.465906] LNet: Removed LNI 192.168.204.158@tcp29 [ 7508.467424] LNet: Removed LNI 192.168.204.158@tcp30 [ 7508.468984] LNet: Removed LNI 192.168.204.158@tcp31 [ 7508.470482] LNet: Removed LNI 192.168.204.158@tcp32 [ 7508.471679] LNet: Removed LNI 192.168.204.158@tcp33 [ 7508.473053] LNet: Removed LNI 192.168.204.158@tcp34 [ 7508.474450] LNet: Removed LNI 192.168.204.158@tcp35 [ 7508.475802] LNet: Removed LNI 192.168.204.158@tcp36 [ 7508.477645] LNet: Removed LNI 192.168.204.158@tcp37 [ 7508.479196] LNet: Removed LNI 192.168.204.158@tcp38 [ 7508.481197] LNet: Removed LNI 192.168.204.158@tcp39 [ 7508.483516] LNet: Removed LNI 192.168.204.158@tcp40 [ 7508.485419] LNet: Removed LNI 192.168.204.158@tcp41 [ 7508.486834] LNet: Removed LNI 192.168.204.158@tcp42 [ 7508.488256] LNet: Removed LNI 192.168.204.158@tcp43 [ 7508.490203] LNet: Removed LNI 192.168.204.158@tcp44 [ 7508.491850] LNet: Removed LNI 192.168.204.158@tcp45 [ 7508.493786] LNet: Removed LNI 192.168.204.158@tcp46 [ 7508.495665] LNet: Removed LNI 192.168.204.158@tcp47 [ 7508.497818] LNet: Removed LNI 192.168.204.158@tcp48 [ 7508.498964] LNet: Removed LNI 192.168.204.158@tcp49 [ 7508.505454] LNet: Removed LNI 192.168.204.158@tcp50 [ 7508.801189] Key type .llcrypt unregistered [ 7508.802308] Key type ._llcrypt unregistered [ 7516.524864] Key type ._llcrypt registered [ 7516.525942] Key type .llcrypt registered [ 7516.556380] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 7516.845916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7516.858741] alg: No test for adler32 (adler32-zlib) [ 7517.700268] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 7517.772567] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 7519.351157] Key type lgssc registered [ 7519.672577] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7521.994421] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7521.999759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7523.157281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7524.430321] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7527.259465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7527.390520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7528.693672] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7530.488656] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7531.669550] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7533.844976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7533.951828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7535.880940] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7537.707877] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7548.326660] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7556.809185] loop0: detected capacity change from 0 to 204800000 [ 7556.869480] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7556.874134] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7556.878677] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7558.952975] loop0: detected capacity change from 0 to 204800000 [ 7559.018698] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7559.024133] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7559.030370] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7560.664370] loop0: detected capacity change from 0 to 204800000 [ 7562.497117] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7562.552621] Lustre: fs15246-OST0000: new disk, initializing [ 7562.554585] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7562.575086] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7564.412209] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7568.208326] loop1: detected capacity change from 0 to 204800000 [ 7569.925597] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7569.960465] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7569.983357] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7570.014548] Lustre: fs15246-MDT0000: new disk, initializing [ 7570.059522] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7570.069185] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7570.107259] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7571.167792] Lustre: 476791:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000b1fb77ec x1858580796963968/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1772480841 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7571.285430] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7580.135259] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7580.255235] LustreError: 476791:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 7580.259885] LustreError: 476791:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 7580.264051] LustreError: 476791:0:(osp_precreate.c:459:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 7584.279640] Lustre: server umount fs15246-MDT0000 complete [ 7585.253331] LustreError: 473768:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7585.260056] LustreError: 473768:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7587.489969] Lustre: server umount fs15246-OST0000 complete [ 7594.464244] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7594.464272] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7594.466335] LustreError: Skipped 1 previous similar message [ 7594.471562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7594.475125] Lustre: Skipped 1 previous similar message [ 7595.489584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7595.493399] Lustre: Skipped 1 previous similar message [ 7600.611375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7600.614706] Lustre: Skipped 1 previous similar message [ 7605.215119] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7605.271476] Lustre: server umount lustre-OST0000 complete [ 7609.311884] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7609.315151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7609.319426] Lustre: Skipped 1 previous similar message [ 7609.322052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7610.848979] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7612.632363] Lustre: server umount lustre-MDT0000 complete [ 7613.957253] LustreError: 473124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772480867 with bad export cookie 10497880549733948938 [ 7613.959307] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7613.961482] LustreError: 473124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7614.087885] Lustre: server umount lustre-MDT0001 complete [ 7617.914565] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 7619.013655] Key type lgssc unregistered [ 7619.152427] LNet: 479853:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7619.155244] LNetError: 479853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7619.168394] LNet: Removed LNI 192.168.204.158@tcp [ 7619.517297] Key type .llcrypt unregistered [ 7619.518301] Key type ._llcrypt unregistered [ 7626.736812] Key type ._llcrypt registered [ 7626.738106] Key type .llcrypt registered [ 7626.784088] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 7627.281561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7627.291420] alg: No test for adler32 (adler32-zlib) [ 7628.141389] Lustre: Lustre: Build Version: 2.17.50_196_gee1d501 [ 7628.216982] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 7629.791102] Key type lgssc registered [ 7630.135965] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7642.329224] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 7645.719124] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 7657.940842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7657.946352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7659.040179] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7659.057886] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7659.099927] Lustre: lustre-MDT0000: new disk, initializing [ 7659.128264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7659.135903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7660.521770] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7664.836355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7664.893331] Lustre: 484236:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7664.925346] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7664.927835] Lustre: Skipped 1 previous similar message [ 7664.974232] Lustre: lustre-MDT0001: new disk, initializing [ 7665.018305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7665.033888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7665.038126] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7666.370830] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7669.509151] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7670.828535] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7673.142195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7673.248539] Lustre: lustre-OST0000: new disk, initializing [ 7673.251382] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7673.282303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7675.140486] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7676.536947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7676.540059] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7676.570619] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7678.180829] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7679.513866] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7679.581673] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7680.851274] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7680.926228] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7682.017441] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7682.019883] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7682.022667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7682.023813] Lustre: Skipped 1 previous similar message [ 7687.139495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7687.142953] Lustre: Skipped 2 previous similar messages [ 7687.702083] Lustre: server umount lustre-OST0000 complete [ 7690.719884] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7690.722374] LustreError: Skipped 1 previous similar message [ 7690.723636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7690.727935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7692.257082] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7695.068997] Lustre: server umount lustre-MDT0000 complete [ 7696.290163] LustreError: 484227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772480950 with bad export cookie 7931822889467657425 [ 7696.293131] LustreError: 484227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7696.294171] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7696.415387] Lustre: server umount lustre-MDT0001 complete [ 7704.407351] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 7707.879925] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 7718.384967] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7449 sec ========== 14:49:30 (1772480970) [ 7718.974942] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:49:31 (1772480971) === [ 7720.139463] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:49:32 (1772480972) === [ 7733.579577] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 7734.635475] Key type lgssc unregistered [ 7734.774396] LNet: 492411:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7734.777600] LNetError: 492411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7734.788346] LNet: Removed LNI 192.168.204.158@tcp [ 7735.100121] Key type .llcrypt unregistered [ 7735.102076] Key type ._llcrypt unregistered