[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 498614309 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003177] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008524] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011142] LSM: Security Framework initializing [ 0.012056] Yama: becoming mindful. [ 0.013042] SELinux: Initializing. [ 0.014069] *** VALIDATE selinux *** [ 0.022230] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027850] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028178] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029118] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030127] *** VALIDATE tmpfs *** [ 0.031505] *** VALIDATE proc *** [ 0.033242] *** VALIDATE cgroup *** [ 0.034012] *** VALIDATE cgroup2 *** [ 0.035284] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036193] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038029] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.042618] debug: unmapping init [mem 0xffffffffa5259000-0xffffffffa5260fff] [ 0.044962] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045672] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046021] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048015] ... generic registers: 4 [ 0.049015] ... value mask: 0000ffffffffffff [ 0.050015] ... max period: 00007fffffffffff [ 0.051013] ... fixed-purpose events: 3 [ 0.052013] ... event mask: 000000070000000f [ 0.053290] rcu: Hierarchical SRCU implementation. [ 0.055459] smp: Bringing up secondary CPUs ... [ 0.056594] x86: Booting SMP configuration: [ 0.057026] .... node #0, CPUs: #1 #2 #3 [ 0.061340] smp: Brought up 1 node, 4 CPUs [ 0.063019] smpboot: Max logical packages: 1 [ 0.064016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.236197] node 0 deferred pages initialised in 170ms [ 0.239525] devtmpfs: initialized [ 0.241280] x86/mm: Memory block size: 128MB [ 0.243759] gcov: version magic: 0x41383552 [ 0.247288] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.248085] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.249333] pinctrl core: initialized pinctrl subsystem [ 0.251157] [ 0.251905] ************************************************************* [ 0.255018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.258016] ** ** [ 0.261017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.264017] ** ** [ 0.267017] ** This means that this kernel is built to expose internal ** [ 0.270017] ** IOMMU data structures, which may compromise security on ** [ 0.273017] ** your system. ** [ 0.275013] ** ** [ 0.277013] ** If you see this message and you are not debugging the ** [ 0.279021] ** kernel, report this immediately to your vendor! ** [ 0.281012] ** ** [ 0.283018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.286014] ************************************************************* [ 0.288344] NET: Registered protocol family 16 [ 0.289428] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.292060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.294050] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.298040] cpuidle: using governor menu [ 0.299698] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.301596] PCI: Using configuration type 1 for base access [ 0.304192] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.315082] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.316019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.317121] cryptd: max_cpu_qlen set to 1000 [ 0.320276] ACPI: Added _OSI(Module Device) [ 0.322014] ACPI: Added _OSI(Processor Device) [ 0.323020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.325014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.329962] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.336578] ACPI: Interpreter enabled [ 0.338055] ACPI: PM: (supports S0 S3 S4 S5) [ 0.339010] ACPI: Using IOAPIC for interrupt routing [ 0.341091] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.344395] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.353551] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.356037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.358017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.362075] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.366372] acpiphp: Slot [2] registered [ 0.368155] acpiphp: Slot [5] registered [ 0.369161] acpiphp: Slot [6] registered [ 0.371146] acpiphp: Slot [7] registered [ 0.372130] acpiphp: Slot [8] registered [ 0.373119] acpiphp: Slot [9] registered [ 0.375122] acpiphp: Slot [10] registered [ 0.376109] acpiphp: Slot [3] registered [ 0.377078] acpiphp: Slot [4] registered [ 0.379083] acpiphp: Slot [11] registered [ 0.380132] acpiphp: Slot [12] registered [ 0.382112] acpiphp: Slot [13] registered [ 0.384113] acpiphp: Slot [14] registered [ 0.385168] acpiphp: Slot [15] registered [ 0.387105] acpiphp: Slot [16] registered [ 0.389110] acpiphp: Slot [17] registered [ 0.390165] acpiphp: Slot [18] registered [ 0.392109] acpiphp: Slot [19] registered [ 0.394106] acpiphp: Slot [20] registered [ 0.395094] acpiphp: Slot [21] registered [ 0.396163] acpiphp: Slot [22] registered [ 0.398103] acpiphp: Slot [23] registered [ 0.400125] acpiphp: Slot [24] registered [ 0.401100] acpiphp: Slot [25] registered [ 0.403099] acpiphp: Slot [26] registered [ 0.404179] acpiphp: Slot [27] registered [ 0.406105] acpiphp: Slot [28] registered [ 0.408104] acpiphp: Slot [29] registered [ 0.409101] acpiphp: Slot [30] registered [ 0.411144] acpiphp: Slot [31] registered [ 0.412087] PCI host bridge to bus 0000:00 [ 0.413022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.416023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.418022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.421025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.424024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.427028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.429175] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.432091] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.436067] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.445014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.450055] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.453024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.455015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.457016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.459336] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.461813] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.465049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.468803] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.473013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.485015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.491018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.496415] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.542014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.551015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.570015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.580311] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.587015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.593015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.611018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.621257] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.632014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.645015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.661016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.673000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.679013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.692017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.711024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.722378] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.731017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.742015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.773015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.783301] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.789019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.795035] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.809031] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.822111] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.824458] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.827437] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.829440] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.831236] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.837119] iommu: Default domain type: Passthrough [ 0.839414] SCSI subsystem initialized [ 0.840152] ACPI: bus type USB registered [ 0.842092] usbcore: registered new interface driver usbfs [ 0.844081] usbcore: registered new interface driver hub [ 0.846095] usbcore: registered new device driver usb [ 0.847142] pps_core: LinuxPPS API ver. 1 registered [ 0.849013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.852056] PTP clock support registered [ 0.854186] EDAC MC: Ver: 3.0.0 [ 0.856141] PCI: Using ACPI for IRQ routing [ 0.858996] NetLabel: Initializing [ 0.860011] NetLabel: domain hash size = 128 [ 0.861012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.864081] NetLabel: unlabeled traffic allowed by default [ 0.866123] vgaarb: loaded [ 0.867260] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.869016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.878330] clocksource: Switched to clocksource kvm-clock [ 1.000281] VFS: Disk quotas dquot_6.6.0 [ 1.001971] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.004809] *** VALIDATE ramfs *** [ 1.006298] *** VALIDATE hugetlbfs *** [ 1.008074] pnp: PnP ACPI init [ 1.010603] pnp: PnP ACPI: found 6 devices [ 1.033318] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.036895] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.038810] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.040769] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.042795] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.044836] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.047352] NET: Registered protocol family 2 [ 1.049756] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.054183] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.057295] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.062232] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.065351] TCP: Hash tables configured (established 65536 bind 65536) [ 1.067972] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.070490] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.072700] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.075181] NET: Registered protocol family 1 [ 1.078266] RPC: Registered named UNIX socket transport module. [ 1.079936] RPC: Registered udp transport module. [ 1.081252] RPC: Registered tcp transport module. [ 1.082443] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.084287] NET: Registered protocol family 44 [ 1.085510] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.087211] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.088744] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.090644] PCI: CLS 0 bytes, default 64 [ 1.091886] Unpacking initramfs... [ 2.243080] hrtimer: interrupt took 10097489 ns [ 3.427767] debug: unmapping init [mem 0xffff9acdbcc54000-0xffff9acdbffbffff] [ 3.432835] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.435966] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.440025] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.504776] Initialise system trusted keyrings [ 5.506868] Key type blacklist registered [ 5.513202] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.527405] zbud: loaded [ 5.533976] *** VALIDATE nfs *** [ 5.535809] *** VALIDATE nfs4 *** [ 5.538054] pstore: using deflate compression [ 5.549171] Platform Keyring initialized [ 5.706205] NET: Registered protocol family 38 [ 5.708150] Key type asymmetric registered [ 5.710634] Asymmetric key parser 'x509' registered [ 5.712968] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.717243] io scheduler mq-deadline registered [ 5.719057] io scheduler kyber registered [ 5.720827] io scheduler bfq registered [ 5.726438] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.729778] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.732970] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.737169] ACPI: Power Button [PWRF] [ 5.745922] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.753570] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.782702] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.807633] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.834037] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.873840] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.914877] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.921721] Non-volatile memory driver v1.3 [ 5.926488] Linux agpgart interface v0.103 [ 6.000311] virtio_blk virtio1: [vda] 134784 512-byte logical blocks (69.0 MB/65.8 MiB) [ 6.006929] vda: detected capacity change from 0 to 69009408 [ 6.052949] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.059905] vdb: detected capacity change from 0 to 1073741824 [ 6.106969] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.122942] vdc: detected capacity change from 0 to 2621440000 [ 6.179761] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.184342] vdd: detected capacity change from 0 to 2621440000 [ 6.219877] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.226409] vde: detected capacity change from 0 to 4294967296 [ 6.273610] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.279107] vdf: detected capacity change from 0 to 4294967296 [ 6.292336] libphy: Fixed MDIO Bus: probed [ 6.306754] usbcore: registered new interface driver usbserial_generic [ 6.310789] usbserial: USB Serial support registered for generic [ 6.314420] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.319906] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.322894] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.326921] mousedev: PS/2 mouse device common for all mice [ 6.330386] rtc_cmos 00:05: RTC can wake from S4 [ 6.336406] rtc_cmos 00:05: registered as rtc0 [ 6.342968] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.352441] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.357463] intel_pstate: CPU model not supported [ 6.359487] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.383599] hid: raw HID events driver (C) Jiri Kosina [ 6.393818] usbcore: registered new interface driver usbhid [ 6.400330] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.401357] usbhid: USB HID core driver [ 6.401556] drop_monitor: Initializing network drop monitor service [ 6.421028] Initializing XFRM netlink socket [ 6.428311] NET: Registered protocol family 10 [ 6.439378] Segment Routing with IPv6 [ 6.440995] NET: Registered protocol family 17 [ 6.446133] mpls_gso: MPLS GSO support [ 6.461624] RAS: Correctable Errors collector initialized. [ 6.466075] AVX version of gcm_enc/dec engaged. [ 6.472841] AES CTR mode by8 optimization enabled [ 6.673252] sched_clock: Marking stable (6673203675, 0)->(7640810476, -967606801) [ 6.685216] registered taskstats version 1 [ 6.689742] Loading compiled-in X.509 certificates [ 6.694284] zswap: loaded using pool lzo/zbud [ 6.768266] Key type big_key registered [ 6.789622] Key type encrypted registered [ 6.792479] ima: No TPM chip found, activating TPM-bypass! [ 6.796207] ima: Allocated hash algorithm: sha1 [ 6.798595] ima: No architecture policies found [ 6.802211] evm: Initialising EVM extended attributes: [ 6.807641] evm: security.selinux [ 6.810030] evm: security.ima [ 6.811900] evm: security.capability [ 6.814253] evm: HMAC attrs: 0x1 [ 6.820794] rtc_cmos 00:05: setting system clock to 2026-05-16 09:12:09 UTC (1778922729) [ 6.836562] debug: unmapping init [mem 0xffffffffa6203000-0xffffffffa63fffff] [ 6.864230] debug: unmapping init [mem 0xffffffffa4f82000-0xffffffffa5258fff] [ 6.878259] Write protecting the kernel read-only data: 28672k [ 6.883250] debug: unmapping init [mem 0xffffffffa3603000-0xffffffffa37fffff] [ 6.899352] debug: unmapping init [mem 0xffffffffa3f14000-0xffffffffa3ffffff] [ 7.016657] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.035969] systemd[1]: Detected virtualization kvm. [ 7.038052] systemd[1]: Detected architecture x86-64. [ 7.042255] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.092566] systemd[1]: No hostname configured. [ 7.097907] systemd[1]: Set hostname to . [ 7.103113] random: systemd: uninitialized urandom read (16 bytes read) [ 7.111324] systemd[1]: Initializing machine ID from random generator. [ 7.429891] random: systemd: uninitialized urandom read (16 bytes read) [ 7.439580] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 7.449456] random: systemd: uninitialized urandom read (16 bytes read) [ 7.463705] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 7.474557] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. Starting Setup Virtual Console... Starting Journal Service... [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.838414] device-mapper: uevent: version 1.0.3 [ 8.847695] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 10.557157] virtio_net virtio0 ens2: renamed from eth0 [ 10.629657] random: fast init done [ 10.884512] scsi host0: ata_piix [ 11.009864] scsi host1: ata_piix [ 11.013130] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 11.022936] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.326510] random: crng init done [ 16.331195] random: 7 urandom warning(s) missed due to ratelimiting [ 16.988682] dracut-initqueue[590]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 18.369702] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.579739] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.463269] SELinux: Disabled at runtime. [ 21.594925] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.610865] systemd[1]: Detected virtualization kvm. [ 21.612814] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 23.461659] systemd[1]: initrd-switch-root.service: Succeeded. [ 23.468706] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 23.476066] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 23.482230] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 23.494051] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 23.512784] systemd[1]: Starting Journal Service... Starting Journal Service... [ 23.523056] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ 24.110701] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 24.895390] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 25.775471] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 25.848625] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 26.119268] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 26.237089] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 30.844319] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 31.465783] NFS: Registering the id_resolver key type [ 31.468483] Key type id_resolver registered [ 31.470587] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg316-server login: [ 83.479148] libcfs: loading out-of-tree module taints kernel. [ 83.520783] Key type ._llcrypt registered [ 83.528134] Key type .llcrypt registered [ 83.685235] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 100.765922] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 102.560256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 102.583636] alg: No test for adler32 (adler32-zlib) [ 104.147529] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 105.294514] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 107.144211] Key type lgssc registered [ 109.189896] Lustre: Echo OBD driver; http://www.lustre.org/ [ 126.574361] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 161.262295] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 171.976681] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 172.027837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 173.445320] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 173.501245] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 173.620211] Lustre: lustre-MDT0000: new disk, initializing [ 173.717884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 173.743779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 177.529270] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 189.714465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 189.786354] Lustre: 6509:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 189.827937] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 189.832058] Lustre: Skipped 1 previous similar message [ 189.971290] Lustre: lustre-MDT0001: new disk, initializing [ 190.063397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 190.104563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 190.119618] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 194.071583] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 198.370160] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 206.902307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 207.124770] Lustre: lustre-OST0000: new disk, initializing [ 207.135287] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 207.146633] Lustre: 8411:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 207.234145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 208.966446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 208.977325] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 209.077829] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 213.897178] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.314068] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 230.500876] Lustre: lustre-OST0001: new disk, initializing [ 230.505573] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 230.524687] Lustre: 9467:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 230.617603] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 237.085258] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.606383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 237.615531] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 237.675362] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 248.806727] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 258.881738] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 264.706803] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing check_logdir /tmp/testlogs/ [ 269.640672] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing yml_node [ 273.643363] Lustre: DEBUG MARKER: Client: 2.17.53.3 [ 275.758122] Lustre: DEBUG MARKER: MDS: 2.17.53.3 [ 278.585809] Lustre: DEBUG MARKER: OSS: 2.17.53.3 [ 280.320216] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat May 16 05:16:41 EDT 2026 [ 301.217807] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 303.128240] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 323.557649] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 323.571730] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 323.599702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 324.581428] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 324.585417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 324.615110] Lustre: Skipped 1 previous similar message [ 324.634212] Lustre: Skipped 1 previous similar message [ 326.922143] Lustre: server umount lustre-MDT0000 complete [ 331.545307] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778923054 with bad export cookie 4822949348176159120 [ 331.549537] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 331.572323] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 334.825159] LustreError: 12095:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 334.825719] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 334.828640] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 334.828648] Lustre: Skipped 2 previous similar messages [ 334.838941] LustreError: 12095:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 6 previous similar messages [ 334.859030] Lustre: Skipped 2 previous similar messages [ 336.867045] LustreError: 6515:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 336.868180] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 336.893078] LustreError: 6515:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 336.909472] Lustre: Skipped 2 previous similar messages [ 337.834746] Lustre: server umount lustre-MDT0001 complete [ 342.953632] Lustre: server umount lustre-OST0000 complete [ 348.135832] Lustre: server umount lustre-OST0001 complete [ 355.594100] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 366.233677] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 403.343297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 403.654605] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 403.686802] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 403.825857] Lustre: lustre-MDT0000: new disk, initializing [ 403.948722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 404.013296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 408.875954] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 420.740881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 420.916757] Lustre: 14983:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 421.001452] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 421.006440] Lustre: Skipped 1 previous similar message [ 421.115695] Lustre: lustre-MDT0001: new disk, initializing [ 421.229956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 421.278330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 421.290486] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 426.209441] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 433.710672] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 438.389775] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 445.058247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 445.397674] Lustre: lustre-OST0000: new disk, initializing [ 445.400813] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 445.410703] Lustre: 16727:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 445.535807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 447.276515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 447.293585] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 447.477238] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 451.437683] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 459.224454] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 464.354195] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 464.710894] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 470.453708] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 470.695761] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 473.060821] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 473.062681] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 473.074446] LustreError: Skipped 1 previous similar message [ 473.080839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 473.106269] Lustre: Skipped 1 previous similar message [ 483.297753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 483.305646] Lustre: Skipped 4 previous similar messages [ 487.392634] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 487.668870] Lustre: server umount lustre-OST0000 complete [ 493.026136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 493.041557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 497.585806] Lustre: server umount lustre-MDT0000 complete [ 498.660651] LustreError: 15850:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 498.677357] LustreError: 15850:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 502.207052] LustreError: 14976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778923224 with bad export cookie 4822949348176160653 [ 502.224872] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 502.854408] Lustre: server umount lustre-MDT0001 complete [ 507.819830] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 05:20:28 (1778923228) [ 510.347125] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 512.222447] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 05:20:33 (1778923233) [ 518.904611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 519.370486] LustreError: 18989:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 519.450715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 523.774556] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 524.779970] LustreError: 18990:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 533.496060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 539.237961] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 546.632199] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 550.521473] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 557.823684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 564.258895] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 570.280413] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 579.556954] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 579.570244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 579.579320] Lustre: Skipped 1 previous similar message [ 579.588911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 579.597277] Lustre: Skipped 2 previous similar messages [ 583.784286] Lustre: server umount lustre-OST0000 complete [ 590.312858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 590.326759] LustreError: Skipped 1 previous similar message [ 593.770760] Lustre: server umount lustre-MDT0000 complete [ 595.937464] LustreError: 18990:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 595.967806] LustreError: 18990:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 597.614113] LustreError: 18969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778923320 with bad export cookie 4822949348176161528 [ 597.620881] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 597.629016] LustreError: 18969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 598.003335] Lustre: server umount lustre-MDT0001 complete [ 608.379683] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 611.148293] Key type lgssc unregistered [ 611.479328] LNet: 22163:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 611.488620] LNetError: 22163:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 611.505876] LNet: Removed LNI 192.168.203.116@tcp [ 612.504138] Key type .llcrypt unregistered [ 612.510832] Key type ._llcrypt unregistered [ 634.173795] Key type ._llcrypt registered [ 634.175499] Key type .llcrypt registered [ 634.287287] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 635.441094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 635.547199] alg: No test for adler32 (adler32-zlib) [ 636.633496] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 636.927725] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 638.624493] Key type lgssc registered [ 640.056207] Lustre: Echo OBD driver; http://www.lustre.org/ [ 648.192285] loop: module loaded [ 648.250315] loop0: detected capacity change from 0 to 204800000 [ 648.447363] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 648.463196] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 648.705854] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 654.005639] loop0: detected capacity change from 0 to 204800000 [ 654.171154] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 654.192743] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 654.207908] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 659.265934] loop0: detected capacity change from 0 to 204800000 [ 664.658765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 664.682936] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 665.941234] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 666.009224] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 666.082234] Lustre: lustre-MDT0000: new disk, initializing [ 666.175497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 666.194795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 670.193772] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 678.358326] loop1: detected capacity change from 0 to 204800000 [ 683.183060] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 683.426376] Lustre: lustre-OST07c6: new disk, initializing [ 683.431043] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 683.435615] Lustre: Skipped 1 previous similar message [ 683.443697] Lustre: 25318:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 683.539035] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 689.134794] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 692.281778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 692.316442] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 692.346813] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 694.519099] Lustre: 24324:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 694.525898] Lustre: 24324:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 694.531027] Lustre: 24324:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 694.536035] Lustre: 24324:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 694.541740] Lustre: 24324:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 694.546250] Lustre: 24324:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 709.568694] Lustre: 24325:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 709.580619] Lustre: 24325:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 709.600987] Lustre: 24325:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 709.614130] Lustre: 24325:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 709.628501] Lustre: 24325:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 709.638276] Lustre: 24325:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 714.762049] Lustre: 24324:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 714.768197] Lustre: 24324:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 714.771662] Lustre: 24324:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 714.776490] Lustre: 24324:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 714.782142] Lustre: 24324:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 714.790183] Lustre: 24324:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 714.796855] Lustre: 24324:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 714.802812] Lustre: 24324:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 714.808054] Lustre: 24324:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 714.812830] Lustre: 24324:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 714.817804] Lustre: 24324:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 714.824033] Lustre: 24324:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 718.019348] Lustre: 24324:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 718.034300] Lustre: 24324:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 718.048312] Lustre: 24324:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 718.061805] Lustre: 24324:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 718.070565] Lustre: 24324:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 718.076250] Lustre: 24324:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 718.084941] Lustre: 24324:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 718.101328] Lustre: 24324:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 718.112326] Lustre: 24324:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 718.130568] Lustre: 24324:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 718.138729] Lustre: 24324:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 718.144807] Lustre: 24324:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 724.603490] Lustre: 25905:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 724.613611] Lustre: 25905:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 724.621448] Lustre: 25905:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 724.625576] Lustre: 25905:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 724.632312] Lustre: 25905:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 724.638392] Lustre: 25905:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 724.643937] Lustre: 25905:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 724.650864] Lustre: 25905:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 724.657179] Lustre: 25905:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 724.665625] Lustre: 25905:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 724.675427] Lustre: 25905:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 724.700558] Lustre: 25905:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 732.607968] Lustre: 24324:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 732.622060] Lustre: 24324:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 732.631835] Lustre: 24324:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 732.638830] Lustre: 24324:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 732.647978] Lustre: 24324:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 732.658164] Lustre: 24324:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 732.663166] Lustre: 24324:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 732.671746] Lustre: 24324:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 732.687637] Lustre: 24324:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 732.697034] Lustre: 24324:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 732.702870] Lustre: 24324:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 732.709606] Lustre: 24324:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 784.356986] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 784.369277] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 784.382762] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 787.649701] Lustre: server umount lustre-OST07c6 complete [ 798.021503] Lustre: server umount lustre-MDT0000 complete [ 812.923869] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 815.937289] Key type lgssc unregistered [ 816.211284] LNet: 27841:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 816.220811] LNetError: 27841:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 816.234301] LNet: Removed LNI 192.168.203.116@tcp [ 817.125599] Key type .llcrypt unregistered [ 817.130700] Key type ._llcrypt unregistered [ 834.753725] Key type ._llcrypt registered [ 834.762680] Key type .llcrypt registered [ 834.934347] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 836.040065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 836.124040] alg: No test for adler32 (adler32-zlib) [ 837.147323] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 837.398711] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 839.096346] Key type lgssc registered [ 840.016684] Lustre: Echo OBD driver; http://www.lustre.org/ [ 847.284074] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 847.314423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 848.813859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 852.852206] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 859.114726] LustreError: 29092:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 859.138420] LustreError: 29092:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 862.290651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 862.757590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 867.467798] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 874.224515] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 878.791675] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 887.605311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 888.172628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 893.386240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 895.710787] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 901.982278] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 919.021170] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 919.034216] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 919.040227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 919.057464] Lustre: Skipped 1 previous similar message [ 922.976974] Lustre: server umount lustre-OST0000 complete [ 929.250039] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 929.257722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 929.277675] Lustre: Skipped 1 previous similar message [ 929.278152] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 929.278156] LustreError: Skipped 1 previous similar message [ 929.326912] Lustre: Skipped 2 previous similar messages [ 933.214974] Lustre: server umount lustre-MDT0000 complete [ 934.371415] LustreError: 29803:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 934.385428] LustreError: 29803:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 936.911075] LustreError: 29072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778923659 with bad export cookie 2752664389587746741 [ 936.923052] LustreError: 29072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 936.924170] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 937.372209] Lustre: server umount lustre-MDT0001 complete [ 947.686307] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 950.366762] Key type lgssc unregistered [ 950.649881] LNet: 32473:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.657195] LNetError: 32473:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 950.686943] LNet: Removed LNI 192.168.203.116@tcp [ 951.570274] Key type .llcrypt unregistered [ 951.573877] Key type ._llcrypt unregistered [ 970.461366] Key type ._llcrypt registered [ 970.464019] Key type .llcrypt registered [ 970.594429] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 971.639930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 971.708816] alg: No test for adler32 (adler32-zlib) [ 972.881987] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 973.184891] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 974.928192] Key type lgssc registered [ 976.391675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 984.330425] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 984.354560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 986.055970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 990.994483] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 996.339365] LustreError: 33728:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 996.363777] LustreError: 33728:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 999.787830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1000.243479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1004.583760] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1010.371246] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1015.010739] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1022.341755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1022.777037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1029.012921] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1031.996690] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1035.590754] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1045.474059] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1045.491138] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1045.510920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1047.525425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1047.544143] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1047.559432] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1051.436927] Lustre: server umount lustre-OST0000 complete [ 1056.741755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1056.749701] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1056.773700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1056.786945] Lustre: Skipped 1 previous similar message [ 1062.764688] Lustre: server umount lustre-MDT0000 complete [ 1062.885814] LustreError: 33728:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1066.842266] LustreError: 35346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778923789 with bad export cookie 15820372377565541865 [ 1066.844963] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1066.857621] LustreError: 35346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1067.136219] Lustre: server umount lustre-MDT0001 complete [ 1077.230149] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1080.028833] Key type lgssc unregistered [ 1080.311679] LNet: 36905:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1080.329099] LNetError: 36905:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1080.341772] LNet: Removed LNI 192.168.203.116@tcp [ 1081.176408] Key type .llcrypt unregistered [ 1081.181310] Key type ._llcrypt unregistered [ 1103.471274] Key type ._llcrypt registered [ 1103.472966] Key type .llcrypt registered [ 1103.569853] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1104.384155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1104.613448] alg: No test for adler32 (adler32-zlib) [ 1105.770140] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 1106.029066] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1107.696168] Key type lgssc registered [ 1109.163216] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1117.473309] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1117.501632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1119.087517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1124.075946] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1128.422336] LustreError: 38303:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1128.448401] LustreError: 38303:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1133.577262] LustreError: 38304:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1134.361582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1134.762378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1139.627222] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1144.848971] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1148.891399] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1155.431640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1155.809169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1161.242236] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1164.035102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1166.048432] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1173.626303] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.16@tcp (stopping) [ 1174.502732] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1174.511694] LustreError: 38299:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1174.517168] Lustre: Skipped 1 previous similar message [ 1174.544703] LustreError: 38299:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1174.593774] Lustre: server umount lustre-MDT0000 complete [ 1178.107551] LustreError: 38285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778923900 with bad export cookie 11457394534951074154 [ 1178.120207] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1178.124936] LustreError: 38285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1178.738996] LustreError: 40172:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1178.785151] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.16@tcp (stopping) [ 1179.620936] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1179.634513] Lustre: Skipped 1 previous similar message [ 1183.652244] LustreError: 40172:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1183.652488] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1183.678923] LustreError: 40172:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1183.698334] Lustre: Skipped 1 previous similar message [ 1184.485034] Lustre: server umount lustre-MDT0001 complete [ 1205.415766] Lustre: server umount lustre-OST0000 complete [ 1221.121731] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1223.519443] Key type lgssc unregistered [ 1223.822269] LNet: 41805:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1223.833567] LNetError: 41805:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1224.869810] LNet: Removed LNI 192.168.203.116@tcp [ 1226.031757] Key type .llcrypt unregistered [ 1226.035631] Key type ._llcrypt unregistered [ 1255.530589] Key type ._llcrypt registered [ 1255.534776] Key type .llcrypt registered [ 1255.754766] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1258.137698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1258.212485] alg: No test for adler32 (adler32-zlib) [ 1259.328225] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 1259.602847] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1261.328694] Key type lgssc registered [ 1262.972859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1272.174386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1272.234582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1273.901438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1278.268675] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1284.068886] LustreError: 43208:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1284.090089] LustreError: 43208:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1287.417787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1287.867808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1292.585816] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1298.699308] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1302.987740] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1310.284778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1310.755528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1317.745919] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1318.961369] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1324.273328] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1332.354926] Lustre: lustre-OST0000: Not available for connect from 192.168.203.16@tcp (stopping) [ 1333.223746] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1333.229162] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1333.253419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1334.258443] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1334.261764] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1334.262196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1337.449092] LustreError: 44804:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1337.548170] Lustre: server umount lustre-OST0000 complete [ 1359.333346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1359.345071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1359.366715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1359.377381] Lustre: Skipped 1 previous similar message [ 1362.311713] Lustre: server umount lustre-MDT0000 complete [ 1365.990331] LustreError: 44827:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1366.031747] LustreError: 44827:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1366.475762] LustreError: 43187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778924089 with bad export cookie 12077463994123194042 [ 1366.483176] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1366.489292] LustreError: 43187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1372.983635] Lustre: server umount lustre-MDT0001 complete [ 1385.622894] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1388.792295] Key type lgssc unregistered [ 1389.112685] LNet: 46563:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1389.121082] LNetError: 46563:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1389.151810] LNet: Removed LNI 192.168.203.116@tcp [ 1390.014899] Key type .llcrypt unregistered [ 1390.019076] Key type ._llcrypt unregistered [ 1416.864778] Key type ._llcrypt registered [ 1416.867894] Key type .llcrypt registered [ 1416.991259] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1418.419690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1418.476214] alg: No test for adler32 (adler32-zlib) [ 1419.646739] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 1419.998715] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1421.713126] Key type lgssc registered [ 1423.262609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1432.200922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1432.245081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1433.910226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1438.702715] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1444.323119] LustreError: 47964:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1444.357243] LustreError: 47964:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1447.969236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1448.566579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1453.776946] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1459.529049] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1464.249495] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1471.936488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1472.316957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1478.533320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1479.557840] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1486.525861] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1494.051647] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1498.652607] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1511.278966] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-05-16 5h37m11s [ 1512.993837] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1519.588041] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1519.590562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1519.620536] Lustre: Skipped 2 previous similar messages [ 1519.630545] Lustre: Skipped 1 previous similar message [ 1520.274469] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.16@tcp (stopping) [ 1521.311802] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.16@tcp (stopping) [ 1521.326367] Lustre: Skipped 3 previous similar messages [ 1521.563369] Lustre: server umount lustre-MDT0000 complete [ 1524.760276] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1526.771310] LustreError: 47965:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1526.816974] LustreError: 47965:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1532.461990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1532.624796] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1532.907493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1532.987982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1533.927590] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1537.911611] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1541.549637] Lustre: DEBUG MARKER: Wait for df (36078) ... [ 1543.861304] Lustre: DEBUG MARKER: done [ 1550.306557] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1550.314453] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1550.329994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1553.378079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1553.385597] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1554.406913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1554.415353] Lustre: Skipped 2 previous similar messages [ 1555.943755] Lustre: server umount lustre-OST0000 complete [ 1564.641938] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1564.644679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1564.666399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1564.689597] Lustre: Skipped 1 previous similar message [ 1575.392361] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1575.686626] Lustre: server umount lustre-MDT0000 complete [ 1577.957640] LustreError: 48675:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1577.984792] LustreError: 48675:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1581.043296] LustreError: 47964:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1581.061806] LustreError: 47964:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1581.099777] LustreError: 47945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778924303 with bad export cookie 16828110118737801246 [ 1581.107648] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1581.117666] LustreError: 47945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1581.965233] Lustre: server umount lustre-MDT0001 complete [ 1596.102338] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1599.121856] Key type lgssc unregistered [ 1599.535616] LNet: 52517:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1599.549430] LNetError: 52517:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1599.574364] LNet: Removed LNI 192.168.203.116@tcp [ 1600.515938] Key type .llcrypt unregistered [ 1600.518461] Key type ._llcrypt unregistered [ 1643.122739] Key type ._llcrypt registered [ 1643.124702] Key type .llcrypt registered [ 1643.225855] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1644.567708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1644.597182] alg: No test for adler32 (adler32-zlib) [ 1645.709410] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 1646.020838] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1647.728389] Key type lgssc registered [ 1648.770519] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1656.206948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1656.222595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1657.468627] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1657.501576] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1657.728048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1661.428955] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1669.332473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1669.486894] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1669.541058] Lustre: 54712:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1669.883185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1674.896987] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1680.534215] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1684.318094] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1692.066147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1692.294434] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1692.611712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1699.127401] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1701.116895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1705.047396] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1713.722132] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-05-16 5h40m33s [ 1715.844935] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1726.974411] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1730.198332] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1730.270181] LustreError: 54719:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1750.699802] Lustre: lustre-MDT0000: Client 137c4063-9b81-41a4-b3c0-6651a08fe890 (at 192.168.203.16@tcp) reconnecting [ 1770.320260] LustreError: 54719:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1772.678574] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1775.416446] Lustre: DEBUG MARKER: done [ 1783.269320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1783.285099] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1783.300516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1783.309860] LustreError: Skipped 1 previous similar message [ 1783.334254] Lustre: Skipped 1 previous similar message [ 1788.452507] Lustre: server umount lustre-OST0000 complete [ 1792.994218] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1792.999107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1793.022063] Lustre: Skipped 1 previous similar message [ 1793.029725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1794.096982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1794.109643] Lustre: Skipped 2 previous similar messages [ 1799.139338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1799.144563] Lustre: Skipped 1 previous similar message [ 1805.283158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1805.302061] Lustre: Skipped 1 previous similar message [ 1806.304389] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1806.573327] Lustre: server umount lustre-MDT0000 complete [ 1810.555443] LustreError: 54704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778924533 with bad export cookie 11292086503448537806 [ 1810.571477] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1810.578820] LustreError: 54704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1810.999197] Lustre: server umount lustre-MDT0001 complete [ 1821.316274] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1823.916226] Key type lgssc unregistered [ 1824.256556] LNet: 58687:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1824.268028] LNetError: 58687:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1824.283150] LNet: Removed LNI 192.168.203.116@tcp [ 1825.088137] Key type .llcrypt unregistered [ 1825.090697] Key type ._llcrypt unregistered [ 1871.682372] Key type ._llcrypt registered [ 1871.684223] Key type .llcrypt registered [ 1871.775377] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1872.846633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1872.952453] alg: No test for adler32 (adler32-zlib) [ 1874.176430] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 1874.523154] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1876.240255] Key type lgssc registered [ 1877.884656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1881.439151] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1881.458883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1882.757541] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1882.781818] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1882.940845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1883.930590] Lustre: Failing over lustre-MDT0000 [ 1884.296355] Lustre: server umount lustre-MDT0000 complete [ 1891.206931] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 05:43:32 (1778924612) [ 1898.196943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1898.593796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1902.605191] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1911.717250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1911.912490] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1911.967236] Lustre: 61473:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1912.252035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1916.898935] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1922.663285] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1926.852432] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1933.733589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1933.959966] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1934.195277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1940.194602] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1940.793259] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1946.263197] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1952.491715] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1960.930411] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1960.932629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1960.952107] Lustre: Skipped 2 previous similar messages [ 1960.967521] Lustre: Skipped 2 previous similar messages [ 1965.319156] Lustre: server umount lustre-MDT0000 complete [ 1969.174562] LustreError: 61464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778924691 with bad export cookie 4870681454138215940 [ 1969.201341] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1969.202795] LustreError: 61464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1969.722845] Lustre: server umount lustre-MDT0001 complete [ 1972.870533] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1977.810037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1986.457336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1986.528136] Lustre: 60394:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778924693/real 1778924693] req@000000001ba96c5f x1865337638264192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778924709 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1986.559257] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1989.664235] Lustre: 60394:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778924696/real 1778924696] req@0000000006edf211 x1865337638264320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778924712 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1991.648272] Lustre: 60395:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778924698/real 1778924698] req@00000000a1bd5a20 x1865337638264448/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778924714 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1994.153202] LustreError: 63093:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1994.165412] LustreError: 63093:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 1994.244698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1994.372216] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1995.235268] LustreError: 63094:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1996.256398] Lustre: 60395:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778924701/real 1778924701] req@0000000092aca092 x1865337638264704/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778924717 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1997.291594] LustreError: 63093:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1997.320942] LustreError: 63093:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1999.412455] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2000.352129] Lustre: 60393:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778924707/real 1778924707] req@00000000ae60fc94 x1865337638264832/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778924723 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2002.407030] LustreError: 63094:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2002.455364] LustreError: 63094:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2007.535946] LustreError: 64902:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2007.572564] LustreError: 64902:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2009.608824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2010.051725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2014.639847] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2015.233372] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2016.317155] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2020.667727] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2025.432115] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2035.685860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2035.702928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2035.718585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2038.733865] Lustre: server umount lustre-MDT0000 complete [ 2041.834063] LustreError: 65618:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2041.868835] LustreError: 65618:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2042.512260] LustreError: 61464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778924765 with bad export cookie 4870681454138218222 [ 2042.516285] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2042.520313] LustreError: 61464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2042.839176] Lustre: server umount lustre-MDT0001 complete [ 2048.795239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2057.995448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2063.328442] Lustre: 60393:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778924770/real 1778924770] req@000000007b5c3e4f x1865337638293760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778924786 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2063.331507] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2063.351733] Lustre: 60393:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2063.381210] Lustre: Skipped 2 previous similar messages [ 2067.488865] LustreError: 60392:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007b5c3e4f x1865337638294656/t0(0) o250->MGC192.168.203.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2067.925373] LustreError: 63093:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2068.067958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2068.139328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2073.507619] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2084.905940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2090.483888] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2090.781576] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2097.098320] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2101.930368] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2110.957518] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2110.964924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2110.996405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2110.999274] Lustre: Skipped 3 previous similar messages [ 2117.092316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2117.107365] Lustre: Skipped 5 previous similar messages [ 2122.210744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2122.217963] Lustre: Skipped 2 previous similar messages [ 2124.256225] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2124.529984] Lustre: server umount lustre-MDT0000 complete [ 2127.337129] LustreError: 67358:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2127.373456] LustreError: 67358:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 10 previous similar messages [ 2128.632078] LustreError: 64042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778924851 with bad export cookie 4870681454138219601 [ 2128.642192] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2128.656476] LustreError: 64042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2129.098089] Lustre: server umount lustre-MDT0001 complete [ 2136.771189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2140.835264] Lustre: DEBUG MARKER: files compared the same [ 2153.587648] Lustre: server umount lustre-OST0000 complete [ 2173.156178] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 2176.294725] Key type lgssc unregistered [ 2176.554695] LNet: 70391:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2176.563716] LNetError: 70391:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2177.640992] LNet: Removed LNI 192.168.203.116@tcp [ 2178.903235] Key type .llcrypt unregistered [ 2178.905908] Key type ._llcrypt unregistered [ 2210.980687] Key type ._llcrypt registered [ 2210.989791] Key type .llcrypt registered [ 2211.157679] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2213.207829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2213.308267] alg: No test for adler32 (adler32-zlib) [ 2214.541265] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 2214.911898] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 2216.584197] Key type lgssc registered [ 2218.291802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2228.009281] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2228.061627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2229.855911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2235.211816] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug +malloc all [ 2239.970106] LustreError: 71794:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2239.999870] LustreError: 71794:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2244.285310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2244.708770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2249.260594] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug +malloc all [ 2255.733910] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2259.798162] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2268.373525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2268.656090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2275.772035] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug +malloc all [ 2275.879079] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2282.390468] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2296.315331] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2296.334741] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2296.364365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2298.162628] Lustre: server umount lustre-OST0000 complete [ 2306.026274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2306.035363] LustreError: Skipped 1 previous similar message [ 2306.041536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2306.054980] Lustre: Skipped 1 previous similar message [ 2306.060506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2306.066370] Lustre: Skipped 1 previous similar message [ 2308.877200] Lustre: server umount lustre-MDT0000 complete [ 2311.671185] LustreError: 71793:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2311.700981] LustreError: 71793:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2313.038170] LustreError: 74033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778925035 with bad export cookie 2127697819343186248 [ 2313.048471] LustreError: 74033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2313.053623] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2313.421311] Lustre: server umount lustre-MDT0001 complete [ 2325.399102] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 2328.348307] Key type lgssc unregistered [ 2328.684768] LNet: 74971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2328.704802] LNetError: 74971:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2328.718835] LNet: Removed LNI 192.168.203.116@tcp [ 2329.636462] Key type .llcrypt unregistered [ 2329.640355] Key type ._llcrypt unregistered [ 2357.405871] Key type ._llcrypt registered [ 2357.409042] Key type .llcrypt registered [ 2357.501421] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2358.602908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2358.750251] alg: No test for adler32 (adler32-zlib) [ 2359.928902] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 2360.216665] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 2361.904567] Key type lgssc registered [ 2363.652981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2372.059750] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2372.107484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2387.808437] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2392.929126] LustreError: 76353:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.116@tcp: failed processing log, type 4: rc = -110 [ 2419.553084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2426.243486] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2434.000042] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2443.171989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2443.278737] Lustre: *** cfs_fail_loc=706, val=0*** [ 2443.746726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2443.818565] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2448.567615] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2448.878474] LustreError: 76379:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2448.899933] LustreError: 76379:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2453.991464] LustreError: 76378:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2455.015372] LustreError: 77327:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2455.033284] LustreError: 77327:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2459.849305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2460.245229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2465.285514] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2470.973728] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2474.719673] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2479.588079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2479.596705] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2479.612263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2480.612555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2480.622825] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2480.623589] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2483.393910] Lustre: server umount lustre-OST0000 complete [ 2490.850488] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2490.877864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2490.884809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2490.899920] Lustre: Skipped 2 previous similar messages [ 2493.075064] Lustre: server umount lustre-MDT0000 complete [ 2495.985171] LustreError: 77326:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2496.020549] LustreError: 77326:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2497.899347] LustreError: 76360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778925220 with bad export cookie 14213505431561570203 [ 2497.902365] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2497.911973] LustreError: 76360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2498.314511] Lustre: server umount lustre-MDT0001 complete [ 2510.491930] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 2514.522334] Key type lgssc unregistered [ 2514.866404] LNet: 79614:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2514.878395] LNetError: 79614:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2514.911328] LNet: Removed LNI 192.168.203.116@tcp [ 2516.058201] Key type .llcrypt unregistered [ 2516.062931] Key type ._llcrypt unregistered [ 2543.330666] Key type ._llcrypt registered [ 2543.335994] Key type .llcrypt registered [ 2543.528765] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2544.715893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2544.760462] alg: No test for adler32 (adler32-zlib) [ 2545.799582] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 2546.079743] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 2547.794203] Key type lgssc registered [ 2549.049657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2557.791703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2557.814981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2564.342651] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2572.346228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2572.917859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2577.105507] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2578.402027] LustreError: 81686:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2578.426077] LustreError: 81686:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 2582.499981] LustreError: 81689:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2582.517290] LustreError: 81689:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2585.090969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2585.441291] LustreError: 81686:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2585.467701] LustreError: 81686:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2585.478515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2590.700259] LustreError: 81689:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2590.735201] LustreError: 81689:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2591.681504] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2595.811189] LustreError: 81686:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2595.820733] LustreError: 81686:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2599.838749] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2599.857447] Lustre: MGS: Client a0bc8d01-cc24-4d01-abb7-19f0006fc911 (at 0@lo) reconnecting [ 2599.880590] LustreError: 81700:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2600.126969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2600.223690] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2604.355407] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2620.903754] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2620.915190] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2620.924022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2620.962563] Lustre: Skipped 1 previous similar message [ 2624.542558] Lustre: server umount lustre-OST0000 complete [ 2631.137604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2631.140793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2631.159162] Lustre: Skipped 1 previous similar message [ 2631.197927] Lustre: Skipped 2 previous similar messages [ 2634.780743] Lustre: server umount lustre-MDT0000 complete [ 2636.259135] LustreError: 83194:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2636.281115] LustreError: 83194:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 2639.042742] Lustre: server umount lustre-MDT0001 complete [ 2643.413892] Lustre: server umount MGS complete [ 2655.551135] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 05:56:16 (1778925376) [ 2686.259343] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 2696.230470] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2732.957287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2737.418570] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2746.870436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2747.118549] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2747.171968] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2747.268780] Lustre: lustre-MDT0001: new disk, initializing [ 2747.332226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2747.367397] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2747.374933] Lustre: 88779:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2747.379793] LustreError: 88779:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2752.407062] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2763.352243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2763.621356] Lustre: lustre-OST0000: new disk, initializing [ 2763.634110] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2763.665145] Lustre: 89531:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2763.760741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2769.584167] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2772.064926] Lustre: 89936:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000bde8d56c x1865338341859456/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1778925510 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2776.057983] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2784.174564] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2784.182496] Lustre: MGS: Client b797b3e2-964e-4252-ba6c-f7bf84bafe03 (at 0@lo) reconnecting [ 2784.226551] Lustre: 90464:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2784.244788] LustreError: 88765:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2784.258662] LustreError: 88765:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2784.285959] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2784.378146] Lustre: lustre-MDT0000: new disk, initializing [ 2784.535859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2784.576801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2784.672843] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2785.639771] Lustre: 90489:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000ccee5f94 x1865338341865856/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1778925524 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2788.581744] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2792.433202] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2794.735751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2794.758156] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2794.887372] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2819.556369] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2819.569502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2819.577799] Lustre: Skipped 1 previous similar message [ 2819.605961] Lustre: Skipped 1 previous similar message [ 2824.673319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2824.685135] Lustre: Skipped 1 previous similar message [ 2825.636253] Lustre: server umount lustre-OST0000 complete [ 2829.793402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2829.816454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2829.824160] Lustre: Skipped 1 previous similar message [ 2835.846885] Lustre: server umount lustre-MDT0000 complete [ 2845.736373] Lustre: server umount lustre-MDT0001 complete [ 2849.516529] Lustre: server umount MGS complete [ 2859.168782] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 05:59:40 (1778925580) [ 2877.807621] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 2880.946728] Key type lgssc unregistered [ 2881.293579] LNet: 92951:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2881.309168] LNetError: 92951:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2881.342151] LNet: Removed LNI 192.168.203.116@tcp [ 2882.238977] Key type .llcrypt unregistered [ 2882.248184] Key type ._llcrypt unregistered [ 2898.919450] Key type ._llcrypt registered [ 2898.924514] Key type .llcrypt registered [ 2899.056642] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2900.149952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2900.268323] alg: No test for adler32 (adler32-zlib) [ 2901.375726] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 2901.582986] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 2903.232364] Key type lgssc registered [ 2904.146245] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2912.394557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2912.416650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2913.941093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2918.715376] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2923.507542] LustreError: 94154:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2923.540067] LustreError: 94154:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2927.449924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2927.901952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2932.652383] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2938.037373] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2942.218211] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2949.563627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2949.940636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2956.343991] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2959.160314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2962.234759] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2974.693542] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2974.700449] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2974.704096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2974.711313] Lustre: Skipped 1 previous similar message [ 2974.727207] LustreError: Skipped 1 previous similar message [ 2976.693954] Lustre: server umount lustre-OST0000 complete [ 2984.418106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2984.428196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2984.451693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2984.454515] Lustre: Skipped 1 previous similar message [ 2986.717782] Lustre: server umount lustre-MDT0000 complete [ 2989.906457] LustreError: 94134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778925712 with bad export cookie 8152966375239697461 [ 2989.908248] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2989.915351] LustreError: 94134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2990.250378] Lustre: server umount lustre-MDT0001 complete [ 3000.178084] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 3002.781097] Key type lgssc unregistered [ 3003.057885] LNet: 97341:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3003.067048] LNetError: 97341:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3004.138607] LNet: Removed LNI 192.168.203.116@tcp [ 3005.053717] Key type .llcrypt unregistered [ 3005.055312] Key type ._llcrypt unregistered [ 3023.935971] Key type ._llcrypt registered [ 3023.941284] Key type .llcrypt registered [ 3024.094569] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3025.715332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3025.769789] alg: No test for adler32 (adler32-zlib) [ 3026.955920] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 3027.193054] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 3028.880200] Key type lgssc registered [ 3030.122800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3036.369992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3036.377878] LustreError: 98402:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 3036.393376] LustreError: 98409:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 3036.401780] LustreError: 98402:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4982 [ 3036.413360] LustreError: 98402:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3036.417746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3036.421427] LustreError: 98402:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3038.003781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3045.685831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3045.984551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3050.405070] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3056.240256] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing lsmod [ 3058.744662] LustreError: 99570:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 3058.810602] LustreError: 99580:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 3058.817123] LustreError: 99570:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4932 [ 3058.823098] LustreError: 99570:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3058.833053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3058.833470] LustreError: 99570:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3059.232098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3064.290403] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3064.294537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3064.304348] Lustre: Skipped 1 previous similar message [ 3064.317243] Lustre: Skipped 2 previous similar messages [ 3067.932962] Lustre: server umount lustre-MDT0000 complete [ 3069.411912] LustreError: 98446:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3069.429184] LustreError: 98446:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 3070.437242] LustreError: 98446:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3070.453143] LustreError: 98446:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 3071.052198] LustreError: 98427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778925793 with bad export cookie 11881175680191799213 [ 3071.062779] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3071.068420] LustreError: 98427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3072.544304] LustreError: 98877:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c9e8dbdf x1865338846203392/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3072.715890] Lustre: server umount lustre-MDT0001 complete [ 3086.912117] Lustre: server umount lustre-OST0000 complete [ 3094.978471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3095.456396] LustreError: 100647:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3095.543131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3099.510939] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3100.649426] LustreError: 100648:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3105.770358] LustreError: 100647:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3107.877124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3108.254087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3112.952672] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3118.826858] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3123.141357] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3131.858282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3132.340489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3140.335970] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3140.563109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3147.348307] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3155.940435] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3155.940674] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3155.943322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3155.985566] Lustre: Skipped 2 previous similar messages [ 3161.059777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3161.068621] Lustre: Skipped 1 previous similar message [ 3161.743757] Lustre: server umount lustre-OST0000 complete [ 3166.179401] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3166.182071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3166.201794] Lustre: Skipped 1 previous similar message [ 3166.217989] Lustre: Skipped 2 previous similar messages [ 3171.297701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3171.309656] Lustre: Skipped 1 previous similar message [ 3172.047454] Lustre: server umount lustre-MDT0000 complete [ 3176.417873] LustreError: 100644:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3176.447814] LustreError: 100644:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3177.030941] LustreError: 100629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778925899 with bad export cookie 11881175680191799913 [ 3177.031803] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3177.046184] LustreError: 100629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3177.574410] Lustre: server umount lustre-MDT0001 complete [ 3189.720433] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 3192.933549] Key type lgssc unregistered [ 3193.250752] LNet: 103823:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3193.262056] LNetError: 103823:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3193.281800] LNet: Removed LNI 192.168.203.116@tcp [ 3194.266535] Key type .llcrypt unregistered [ 3194.278592] Key type ._llcrypt unregistered [ 3218.621429] Key type ._llcrypt registered [ 3218.624684] Key type .llcrypt registered [ 3218.744657] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3219.986300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3219.998934] alg: No test for adler32 (adler32-zlib) [ 3220.969434] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 3221.209142] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 3222.840175] Key type lgssc registered [ 3224.049726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3231.868782] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3231.920808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3233.471110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3237.691467] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3244.001957] LustreError: 105223:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3244.016980] LustreError: 105223:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 3246.165743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3246.610840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3250.680309] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3256.311186] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3259.848082] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3266.594549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3266.963423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3273.680984] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3279.255297] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3279.299328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3285.111806] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3289.318683] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3294.689434] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3294.693202] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3294.712157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3294.716826] Lustre: Skipped 1 previous similar message [ 3299.816844] Lustre: server umount lustre-OST0000 complete [ 3304.962356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3304.988965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3304.991923] Lustre: Skipped 1 previous similar message [ 3310.050488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3310.078692] Lustre: Skipped 1 previous similar message [ 3310.595199] Lustre: server umount lustre-MDT0000 complete [ 3315.082675] LustreError: 105204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778926037 with bad export cookie 2770408680231504595 [ 3315.092932] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3315.097118] LustreError: 105204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3315.476517] Lustre: server umount lustre-MDT0001 complete [ 3331.103981] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 3334.228425] Key type lgssc unregistered [ 3334.615896] LNet: 108507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3334.628763] LNetError: 108507:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3335.654532] LNet: Removed LNI 192.168.203.116@tcp [ 3336.598946] Key type .llcrypt unregistered [ 3336.604650] Key type ._llcrypt unregistered [ 3355.910842] Key type ._llcrypt registered [ 3355.913958] Key type .llcrypt registered [ 3356.044734] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3357.489991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3357.529534] alg: No test for adler32 (adler32-zlib) [ 3358.550143] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 3358.844826] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 3360.554619] Key type lgssc registered [ 3361.903960] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3369.570718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3369.604618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3371.189640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3375.640724] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3380.770900] LustreError: 109713:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3380.788266] LustreError: 109713:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 3383.970585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3384.358715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3388.835910] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3394.429667] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3398.039343] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3404.643915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3405.004346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3411.624520] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3416.322910] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3418.005989] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3426.787811] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3426.789040] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3426.797512] Lustre: Skipped 1 previous similar message [ 3426.801386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3426.840350] Lustre: Skipped 1 previous similar message [ 3431.913345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3431.925648] Lustre: Skipped 1 previous similar message [ 3432.350592] Lustre: server umount lustre-OST0000 complete [ 3437.027343] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3437.033934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3437.044389] Lustre: Skipped 1 previous similar message [ 3437.063456] Lustre: Skipped 1 previous similar message [ 3442.147463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3442.155619] Lustre: Skipped 1 previous similar message [ 3442.738153] Lustre: server umount lustre-MDT0000 complete [ 3446.712225] LustreError: 109695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778926169 with bad export cookie 14669005176481308082 [ 3446.716944] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3446.721073] LustreError: 109695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3447.329469] Lustre: server umount lustre-MDT0001 complete [ 3458.884412] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 3461.756693] Key type lgssc unregistered [ 3462.038774] LNet: 112888:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3462.046223] LNetError: 112888:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3462.065744] LNet: Removed LNI 192.168.203.116@tcp [ 3462.940314] Key type .llcrypt unregistered [ 3462.942125] Key type ._llcrypt unregistered [ 3485.238144] Key type ._llcrypt registered [ 3485.240716] Key type .llcrypt registered [ 3485.395828] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3486.248901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3486.370089] alg: No test for adler32 (adler32-zlib) [ 3487.492618] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 3487.726829] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 3489.456183] Key type lgssc registered [ 3490.545426] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3497.843426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3497.868343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3499.404217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3503.849283] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3509.738876] LustreError: 114288:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3509.761502] LustreError: 114288:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 3512.089377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3512.334914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3516.846730] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3523.690893] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3528.596486] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3536.567232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3536.943298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3543.921243] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3545.113378] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3550.241421] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3559.038380] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3564.079650] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3564.093037] Lustre: Skipped 2 previous similar messages [ 3567.233831] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3567.281629] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3569.437137] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3569.460812] Lustre: Skipped 2 previous similar messages [ 3573.811815] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3573.818175] Lustre: Skipped 1 previous similar message [ 3582.228107] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.16@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3582.244975] Lustre: Skipped 2 previous similar messages [ 3590.233266] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.16@tcp 192.168.0.[2,10]@tcp [ 3595.545978] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.16/24@tcp in log lustre-MDT0000 [ 3595.565677] Lustre: Skipped 2 previous similar messages [ 3599.573129] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.16/24@tcp [ 3599.579249] Lustre: Skipped 1 previous similar message [ 3610.091047] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3610.092082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3610.104573] LustreError: Skipped 1 previous similar message [ 3610.144584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3611.455033] Lustre: server umount lustre-OST0000 complete [ 3616.738621] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3616.758246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3616.767667] Lustre: Skipped 1 previous similar message [ 3621.718382] Lustre: server umount lustre-MDT0000 complete [ 3621.858373] LustreError: 114288:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3621.879391] LustreError: 114288:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3625.822922] LustreError: 114268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778926348 with bad export cookie 14354482312161148379 [ 3625.828316] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3625.845244] LustreError: 114268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3636.529846] Lustre: server umount lustre-MDT0001 complete [ 3647.434827] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 3650.193815] Key type lgssc unregistered [ 3650.440802] LNet: 118885:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3650.449675] LNetError: 118885:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3650.473449] LNet: Removed LNI 192.168.203.116@tcp [ 3651.359350] Key type .llcrypt unregistered [ 3651.361957] Key type ._llcrypt unregistered [ 3680.834197] Key type ._llcrypt registered [ 3680.836766] Key type .llcrypt registered [ 3680.914842] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3681.842375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3682.004338] alg: No test for adler32 (adler32-zlib) [ 3683.098673] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 3683.374157] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 3685.144156] Key type lgssc registered [ 3686.311908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3693.425754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3693.457076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3694.951301] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3694.975301] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.16/24@tcp [ 3695.030250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3699.530609] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3705.314208] LustreError: 120431:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3705.334993] LustreError: 120431:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 3708.097200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3708.467835] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3708.527624] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.16/24@tcp [ 3708.562640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3712.639413] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3718.183220] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3722.179393] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3729.436957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3729.879055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3736.579299] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3740.155889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3742.058219] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3754.468915] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3754.478533] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3754.505904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3755.490433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3755.496178] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3755.511051] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3758.654761] Lustre: server umount lustre-OST0000 complete [ 3764.709741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3764.736320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3764.774765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3764.777765] Lustre: Skipped 1 previous similar message [ 3769.980716] Lustre: server umount lustre-MDT0000 complete [ 3770.852867] LustreError: 122590:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3770.873703] LustreError: 122590:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3774.240748] LustreError: 120411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778926496 with bad export cookie 760112088729752903 [ 3774.249981] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3774.257843] LustreError: 120411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3774.746911] Lustre: server umount lustre-MDT0001 complete [ 3787.540855] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 3790.637805] Key type lgssc unregistered [ 3791.043621] LNet: 123663:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3791.069689] LNetError: 123663:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3791.126180] LNet: Removed LNI 192.168.203.116@tcp [ 3791.890150] Key type .llcrypt unregistered [ 3791.894248] Key type ._llcrypt unregistered [ 3821.535261] Key type ._llcrypt registered [ 3821.539947] Key type .llcrypt registered [ 3821.667346] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 3838.288671] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3839.873328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3839.906920] alg: No test for adler32 (adler32-zlib) [ 3840.992174] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 3841.238889] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 3842.930975] Key type lgssc registered [ 3844.091687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3878.247948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3878.284985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3879.616744] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3879.655579] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3879.742769] Lustre: lustre-MDT0000: new disk, initializing [ 3879.837886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3879.852870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3883.933557] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3894.119572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3894.233716] Lustre: 127030:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3894.299837] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3894.304825] Lustre: Skipped 1 previous similar message [ 3894.389321] Lustre: lustre-MDT0001: new disk, initializing [ 3894.511496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3894.554498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3894.565247] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3899.026741] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3905.940595] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3910.000342] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3916.616238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3916.943925] Lustre: lustre-OST0000: new disk, initializing [ 3916.948231] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3916.954225] Lustre: 128768:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3917.048863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3922.569738] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3926.590154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3926.602633] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3926.715081] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3929.448717] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3933.634488] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3933.911527] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3937.866043] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3938.079564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3942.378157] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3942.383076] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3942.394396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3942.411783] Lustre: Skipped 1 previous similar message [ 3942.421077] Lustre: Skipped 1 previous similar message [ 3947.490605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3947.503579] Lustre: Skipped 1 previous similar message [ 3952.622183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3952.635014] Lustre: Skipped 1 previous similar message [ 3954.144194] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3954.338602] Lustre: server umount lustre-OST0000 complete [ 3961.314045] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3961.322161] LustreError: Skipped 1 previous similar message [ 3961.331620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3961.357751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3962.850328] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3964.516537] Lustre: server umount lustre-MDT0000 complete [ 3967.738133] LustreError: 127020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778926690 with bad export cookie 17969501534108137928 [ 3967.739982] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3967.752861] LustreError: 127020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3967.986633] Lustre: server umount lustre-MDT0001 complete [ 3976.920927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3977.543667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3982.064772] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3982.821443] LustreError: 130841:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3982.845359] LustreError: 130841:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 3987.946902] LustreError: 130840:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3991.465578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3991.957574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3996.112486] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4001.490782] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4004.958079] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4011.004902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4011.268634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4016.122884] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4021.113935] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4025.251681] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4025.477285] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4030.024927] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4030.322968] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4039.831079] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4040.013867] Lustre: lustre-OST0001: new disk, initializing [ 4040.018475] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4040.026261] Lustre: 133839:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4040.126131] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4046.108822] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4047.420886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4047.431848] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4047.519750] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4053.262942] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4053.467606] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4057.577292] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4057.769519] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4061.736710] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90f0c2f26000.ost_server_uuid 50 [ 4063.177577] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90f0c2f26000.ost_server_uuid in FULL state after 0 sec [ 4072.149479] Lustre: server umount lustre-OST0001 complete [ 4073.444046] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4073.450725] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4073.460511] LustreError: Skipped 1 previous similar message [ 4073.464033] LustreError: 132438:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4073.480215] Lustre: Skipped 1 previous similar message [ 4073.542721] LustreError: 132438:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4078.050948] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4078.065955] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4078.093736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4078.105074] Lustre: Skipped 2 previous similar messages [ 4078.575725] LustreError: 134118:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4078.605208] LustreError: 134118:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4082.499818] Lustre: server umount lustre-OST0000 complete [ 4088.807854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4088.810878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4088.823536] Lustre: Skipped 2 previous similar messages [ 4088.837184] Lustre: Skipped 2 previous similar messages [ 4092.623542] Lustre: server umount lustre-MDT0000 complete [ 4093.922096] LustreError: 130840:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4093.945449] LustreError: 130840:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4096.363335] LustreError: 130820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778926819 with bad export cookie 17969501534108138810 [ 4096.375300] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4096.377566] LustreError: 130820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4096.746255] Lustre: server umount lustre-MDT0001 complete [ 4114.682633] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 4117.535697] Key type lgssc unregistered [ 4117.835623] LNet: 136672:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4117.844383] LNetError: 136672:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4118.887258] LNet: Removed LNI 192.168.203.116@tcp [ 4119.643889] Key type .llcrypt unregistered [ 4119.646114] Key type ._llcrypt unregistered [ 4156.237352] Key type ._llcrypt registered [ 4156.242430] Key type .llcrypt registered [ 4156.353310] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 4170.613440] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4171.554173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4171.798150] alg: No test for adler32 (adler32-zlib) [ 4172.828573] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 4173.024934] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 4174.680198] Key type lgssc registered [ 4175.683690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4204.089046] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4204.113311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4205.345158] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4205.372743] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4205.448439] Lustre: lustre-MDT0000: new disk, initializing [ 4205.588447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4205.606356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4209.593320] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4218.812661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4218.948528] Lustre: 140793:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4219.018598] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4219.022260] Lustre: Skipped 1 previous similar message [ 4219.119556] Lustre: lustre-MDT0001: new disk, initializing [ 4219.217936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4219.319964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4219.332049] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4223.220869] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4229.162137] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4232.950641] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4238.625427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4238.934971] Lustre: lustre-OST0000: new disk, initializing [ 4238.938962] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4238.956804] Lustre: 142532:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4239.051612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4244.102906] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4249.160970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4249.180445] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4249.316361] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4250.948276] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4255.948699] Lustre: Failing over lustre-OST0000 [ 4256.086900] Lustre: server umount lustre-OST0000 complete [ 4272.913556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4273.176153] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4273.190767] Lustre: Skipped 1 previous similar message [ 4273.213963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4273.226723] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4274.144660] Lustre: 139018:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778926980/real 1778926980] req@0000000071176f19 x1865340047882240/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1778926996 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4274.289952] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4276.192112] Lustre: 139018:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778926982/real 1778926982] req@000000003c491474 x1865340047883392/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1778926998 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4277.206742] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4278.280705] Lustre: lustre-OST0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. [ 4278.281254] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4279.750660] Lustre: Failing over lustre-MDT0000 [ 4279.999668] Lustre: server umount lustre-MDT0000 complete [ 4280.800448] Lustre: 139018:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778926987/real 1778926987] req@00000000d93bd85a x1865340047884416/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1778927003 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4280.802953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4280.838184] Lustre: 139018:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4280.852562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4283.366466] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4283.383929] Lustre: Skipped 1 previous similar message [ 4285.408227] Lustre: 139015:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778926992/real 1778926992] req@00000000b6f62bd2 x1865340047885568/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1778927008 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4288.485839] LustreError: 140804:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4288.500181] LustreError: 140804:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 8 previous similar messages [ 4289.630055] LustreError: 140801:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4293.604914] LustreError: 141894:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4293.624181] LustreError: 141894:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4297.164280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4297.184090] Lustre: 140796:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778927003/real 1778927003] req@00000000cd2f9113 x1865340047893376/t0(0) o101->MGC192.168.203.116@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1778927019 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4297.209496] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4297.225654] Lustre: 140796:0:(mgc_request.c:1897:mgc_process_log()) MGC192.168.203.116@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 4297.230933] LustreError: 141894:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4297.264663] LustreError: 141894:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 4302.307760] LustreError: 140799:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4302.322088] LustreError: 140799:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 4306.412826] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x5a2210d98a28920 [ 4306.426915] Lustre: MGC192.168.203.116@tcp: Connection restored to 0@lo (at 0@lo) [ 4306.436526] Lustre: Skipped 1 previous similar message [ 4306.626265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4306.684609] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4309.624656] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4310.232527] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4312.046199] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4312.067995] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4312.095835] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4317.159769] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4317.163260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4317.170951] Lustre: Skipped 1 previous similar message [ 4317.174713] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4317.177601] Lustre: Skipped 1 previous similar message [ 4322.093966] Lustre: server umount lustre-OST0000 complete [ 4327.398379] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4327.399065] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4327.419133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4327.431252] Lustre: Skipped 1 previous similar message [ 4331.579511] Lustre: server umount lustre-MDT0000 complete [ 4332.514883] LustreError: 140804:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4332.545517] LustreError: 140804:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 4334.721722] LustreError: 140783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778927057 with bad export cookie 405923258695846176 [ 4334.727850] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4334.730325] LustreError: 140783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4335.002773] Lustre: server umount lustre-MDT0001 complete [ 4343.392361] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 4345.772931] Key type lgssc unregistered [ 4346.031529] LNet: 146024:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4346.038452] LNetError: 146024:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4346.050257] LNet: Removed LNI 192.168.203.116@tcp [ 4346.655682] Key type .llcrypt unregistered [ 4346.658511] Key type ._llcrypt unregistered [ 4377.320460] Key type ._llcrypt registered [ 4377.324843] Key type .llcrypt registered [ 4377.422039] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4378.314988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4378.360050] alg: No test for adler32 (adler32-zlib) [ 4379.424521] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 4379.604469] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 4381.256518] Key type lgssc registered [ 4382.116492] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4388.523990] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4388.535451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4389.940850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4392.806868] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4398.390759] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4400.099640] LustreError: 147525:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4400.130103] LustreError: 147525:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4401.762071] LustreError: 147553:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4401.769570] LustreError: 147553:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4401.778479] LustreError: 147553:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 12, retries 0, failed: rc = -5 [ 4402.021881] Lustre: server umount lustre-MDT0000 complete [ 4411.868911] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 4418.104910] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4440.489467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4440.683652] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4440.715444] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4440.781600] Lustre: lustre-MDT0000: new disk, initializing [ 4440.832963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4440.851615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4444.028416] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4452.553084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4452.644951] Lustre: 150706:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4452.689128] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4452.692841] Lustre: Skipped 1 previous similar message [ 4452.748985] Lustre: lustre-MDT0001: new disk, initializing [ 4452.822590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4452.857755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4452.867592] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4456.084440] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4461.849613] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4464.509820] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4469.133980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4469.334859] Lustre: lustre-OST0000: new disk, initializing [ 4469.338257] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4469.343432] Lustre: 152445:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4469.417725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4473.227034] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4475.437974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4475.452756] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4475.530961] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4478.479963] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4482.396630] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4483.766754] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 06:26:45 (1778927205) [ 4662.756108] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4662.761531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4662.769900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4665.313116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4665.318509] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4665.334082] Lustre: Skipped 1 previous similar message [ 4667.345165] Lustre: server umount lustre-MDT0000 complete [ 4669.622209] LustreError: 150698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778927392 with bad export cookie 7534468310592225457 [ 4669.623719] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4669.630980] LustreError: 150698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4670.436413] LustreError: 150713:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4670.436772] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4670.467461] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4670.470281] Lustre: Skipped 2 previous similar messages [ 4674.464671] LustreError: 154093:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4674.465270] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4675.893737] Lustre: server umount lustre-MDT0001 complete [ 4684.672590] Lustre: server umount lustre-OST0000 complete [ 4690.502938] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 4697.162850] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4722.953787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4723.116667] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4723.141165] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4723.193989] Lustre: lustre-MDT0000: new disk, initializing [ 4723.254328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4723.271544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4725.795613] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4732.706353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4732.789572] Lustre: 156946:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4732.843428] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4732.847290] Lustre: Skipped 1 previous similar message [ 4732.900258] Lustre: lustre-MDT0001: new disk, initializing [ 4732.951072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4732.974784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4732.983629] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4735.868932] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4740.917438] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4743.261294] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4746.941793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4747.083108] Lustre: lustre-OST0000: new disk, initializing [ 4747.086800] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4747.094661] Lustre: 158684:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4750.380890] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4755.271129] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4755.994605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4756.003323] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4756.081899] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4757.886591] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4758.036923] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4760.401164] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4760.523771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4763.898249] Lustre: server umount lustre-OST0000 complete [ 4766.177866] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4766.180092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4766.186263] Lustre: Skipped 1 previous similar message [ 4766.194666] Lustre: Skipped 1 previous similar message [ 4775.454762] Lustre: server umount lustre-MDT0000 complete [ 4776.418307] LustreError: 156957:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4776.426969] LustreError: 156957:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4777.501546] LustreError: 156936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778927500 with bad export cookie 7534468310592384287 [ 4777.504032] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4777.508677] LustreError: 156936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4787.788978] Lustre: server umount lustre-MDT0001 complete [ 4792.729909] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 06:31:54 (1778927514) [ 4802.215123] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 4807.274545] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4826.914541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4827.068550] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4827.097699] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4827.142401] Lustre: lustre-MDT0000: new disk, initializing [ 4827.201120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4827.208088] Lustre: Skipped 1 previous similar message [ 4827.222862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4829.507726] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4835.428083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4835.515190] Lustre: 163253:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4835.683653] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4837.797793] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4842.101655] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4843.960372] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4847.744345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4847.931356] Lustre: 165001:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4851.028299] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4855.123617] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4855.825948] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4855.879186] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4866.019728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4866.020584] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4866.022693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4866.022698] Lustre: Skipped 2 previous similar messages [ 4866.030968] Lustre: Skipped 1 previous similar message [ 4866.044036] LustreError: Skipped 1 previous similar message [ 4869.492921] Lustre: server umount lustre-OST0000 complete [ 4871.650539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4876.259985] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4876.266295] Lustre: Skipped 1 previous similar message [ 4877.569618] Lustre: server umount lustre-MDT0000 complete [ 4879.412445] LustreError: 163245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778927602 with bad export cookie 7534468310592385176 [ 4879.418623] LustreError: 163245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4879.418846] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4879.584934] Lustre: server umount lustre-MDT0001 complete [ 4883.585498] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 06:33:25 (1778927605) [ 4891.661829] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 4896.310706] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4915.671392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4915.817359] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4915.837181] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4915.840270] Lustre: Skipped 3 previous similar messages [ 4915.882428] Lustre: lustre-MDT0000: new disk, initializing [ 4915.888261] Lustre: Skipped 2 previous similar messages [ 4915.939979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4915.949278] Lustre: Skipped 2 previous similar messages [ 4915.961559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4915.967438] Lustre: Skipped 2 previous similar messages [ 4918.083509] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4923.994262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4924.059652] Lustre: 169167:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4924.197175] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4926.243753] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4930.735404] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4932.708550] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4936.440527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4936.585047] Lustre: 170916:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4939.848750] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4943.945853] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4944.570297] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4948.965034] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4948.965159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4948.970841] LustreError: Skipped 1 previous similar message [ 4948.971684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4948.982573] Lustre: Skipped 1 previous similar message [ 4948.984993] Lustre: Skipped 4 previous similar messages [ 4950.127734] Lustre: server umount lustre-OST0000 complete [ 4958.411323] Lustre: server umount lustre-MDT0000 complete [ 4959.202891] LustreError: 169178:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4959.215818] LustreError: 169178:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4960.544717] LustreError: 169158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778927683 with bad export cookie 7534468310592386261 [ 4960.547259] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4960.553034] LustreError: 169158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4960.791681] Lustre: server umount lustre-MDT0001 complete [ 4966.218785] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 4967.745873] Key type lgssc unregistered [ 4967.937361] LNet: 172637:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4967.944373] LNetError: 172637:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4967.956270] LNet: Removed LNI 192.168.203.116@tcp [ 4968.353432] Key type .llcrypt unregistered [ 4968.355154] Key type ._llcrypt unregistered [ 4981.573308] Key type ._llcrypt registered [ 4981.575212] Key type .llcrypt registered [ 4981.625547] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4982.235304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4982.246828] alg: No test for adler32 (adler32-zlib) [ 4983.154826] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 4983.260596] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 4984.856171] Key type lgssc registered [ 4985.382720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4988.888562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4988.900648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4990.184884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4991.875594] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4995.934568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4996.135285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4997.976411] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5000.735629] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5002.430914] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5005.837745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5006.037975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5009.275811] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5012.073731] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5015.276578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5036.002944] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5036.009306] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5036.010850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5036.016750] Lustre: Skipped 1 previous similar message [ 5039.012911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5039.017299] Lustre: Skipped 1 previous similar message [ 5040.666379] Lustre: server umount lustre-OST0000 complete [ 5044.194755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5044.198517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5044.202367] Lustre: Skipped 2 previous similar messages [ 5044.210198] Lustre: Skipped 1 previous similar message [ 5048.414648] Lustre: server umount lustre-MDT0000 complete [ 5049.313760] LustreError: 174016:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5049.323346] LustreError: 174016:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 5049.983187] LustreError: 173997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778927772 with bad export cookie 8972243632087169270 [ 5049.986073] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5049.988627] LustreError: 173997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5050.121902] Lustre: server umount lustre-MDT0001 complete [ 5054.535944] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 5055.767449] Key type lgssc unregistered [ 5055.921650] LNet: 177218:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5055.925859] LNetError: 177218:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5055.936907] LNet: Removed LNI 192.168.203.116@tcp [ 5056.289278] Key type .llcrypt unregistered [ 5056.291143] Key type ._llcrypt unregistered [ 5067.830673] Key type ._llcrypt registered [ 5067.832545] Key type .llcrypt registered [ 5067.885866] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 5068.207736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5068.219249] alg: No test for adler32 (adler32-zlib) [ 5069.100371] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 5069.195582] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 5070.784226] Key type lgssc registered [ 5071.250098] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5074.629345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5074.640402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5075.885112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5077.763137] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5081.760788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5081.992637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5083.733558] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5086.403550] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5088.110478] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5091.114874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5091.259595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5093.664847] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5095.879096] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5100.459565] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5105.635980] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5105.636295] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5105.638201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5105.643900] Lustre: Skipped 1 previous similar message [ 5106.278511] Lustre: lustre-OST0000: Not available for connect from 192.168.203.16@tcp (stopping) [ 5106.281944] Lustre: Skipped 1 previous similar message [ 5109.664414] Lustre: server umount lustre-OST0000 complete [ 5111.842556] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff90f0c45d7800.ost_server_uuid 50 [ 5112.519373] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5123.041254] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5123.045193] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5123.052147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5126.113990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5126.114763] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5128.858281] Lustre: server umount lustre-MDT0000 complete [ 5130.222355] LustreError: 178575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778927852 with bad export cookie 13686733767070240380 [ 5130.225065] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5130.228337] LustreError: 178575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5130.337381] Lustre: server umount lustre-MDT0001 complete [ 5133.600743] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 06:37:35 (1778927855) [ 5136.473655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5136.719693] LustreError: 182047:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5136.729380] LustreError: 182047:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5136.756551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5138.560810] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5141.986753] LustreError: 182048:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5142.278855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5142.467344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5144.194785] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5146.826809] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5148.487614] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5151.486765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5151.660265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5154.397134] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5156.857781] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5159.861043] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5159.970537] Lustre: lustre-OST0001: new disk, initializing [ 5159.973227] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5159.976327] Lustre: 184544:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5162.095909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5162.867703] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5167.132838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5167.138864] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5167.220027] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5167.410697] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5172.199401] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5172.201862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5172.210806] Lustre: Skipped 1 previous similar message [ 5172.218929] Lustre: Skipped 2 previous similar messages [ 5175.046847] Lustre: server umount lustre-OST0000 complete [ 5177.314214] LustreError: 183639:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5177.321710] LustreError: 183639:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5177.631753] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5177.720854] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5179.294826] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5179.394393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5182.434389] LustreError: 183639:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5182.440843] LustreError: 183639:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5187.557990] LustreError: 184782:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5187.566783] LustreError: 184782:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5192.674416] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5192.681583] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5192.690254] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5192.694977] Lustre: Skipped 2 previous similar messages [ 5196.310090] Lustre: server umount lustre-OST0001 complete [ 5198.818158] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5198.821827] LustreError: Skipped 1 previous similar message [ 5202.913490] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5202.922041] Lustre: Skipped 2 previous similar messages [ 5204.233423] Lustre: server umount lustre-MDT0000 complete [ 5205.988239] LustreError: 184543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778927928 with bad export cookie 13686733767070241437 [ 5205.988966] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5205.994542] LustreError: 184543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5206.137390] Lustre: server umount lustre-MDT0001 complete [ 5216.786639] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 06:38:58 (1778927938) [ 5219.897754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5220.066808] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5220.091983] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5220.227301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5220.231993] Lustre: Skipped 1 previous similar message [ 5222.228571] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5226.243879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5226.312089] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5226.344050] Lustre: 187806:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5228.135478] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5230.479727] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5232.080916] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5234.801193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5234.918160] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5237.581407] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5239.909374] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5241.372165] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5242.804561] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5242.846656] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5245.200236] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5247.554321] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5251.553478] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5251.553734] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5251.556771] LustreError: Skipped 1 previous similar message [ 5251.563868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5251.568604] Lustre: Skipped 4 previous similar messages [ 5263.328124] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5263.370057] Lustre: server umount lustre-OST0000 complete [ 5263.451943] LustreError: 189429:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5263.459879] LustreError: 189429:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 5279.201497] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5279.205595] LustreError: Skipped 1 previous similar message [ 5280.542146] Lustre: server umount lustre-OST0001 complete [ 5284.321766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5284.323133] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5284.324718] Lustre: Skipped 9 previous similar messages [ 5284.330645] Lustre: Skipped 4 previous similar messages [ 5288.164685] Lustre: server umount lustre-MDT0000 complete [ 5289.440971] LustreError: 189344:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5289.448191] LustreError: 189344:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 10 previous similar messages [ 5289.633711] LustreError: 187799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778928012 with bad export cookie 13686733767070242816 [ 5289.636415] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5289.639879] LustreError: 187799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5289.761258] Lustre: server umount lustre-MDT0001 complete [ 5299.257411] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 06:40:21 (1778928021) [ 5305.989878] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 5310.047719] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 5324.603346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5324.718048] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5324.738569] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5324.779614] Lustre: lustre-MDT0000: new disk, initializing [ 5324.823903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5324.827356] Lustre: Skipped 3 previous similar messages [ 5324.837450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5326.444539] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5331.443890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5331.491502] Lustre: 195388:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5331.524678] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5331.528330] Lustre: Skipped 1 previous similar message [ 5331.578648] Lustre: lustre-MDT0001: new disk, initializing [ 5331.649268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5331.655711] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5333.365923] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5337.092162] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5338.694197] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5341.841907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5341.988426] Lustre: lustre-OST0000: new disk, initializing [ 5341.991029] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5341.994673] Lustre: 197127:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5344.597614] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5347.329955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5347.337370] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5347.391307] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5348.627547] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5350.496942] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5350.594228] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5352.494858] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5352.614332] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5357.539892] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5357.541178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5357.549659] LustreError: Skipped 2 previous similar messages [ 5357.557316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5357.563178] Lustre: Skipped 2 previous similar messages [ 5359.667873] Lustre: server umount lustre-OST0000 complete [ 5367.484396] Lustre: server umount lustre-MDT0000 complete [ 5367.780915] LustreError: 195396:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5367.792799] LustreError: 195396:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5369.280422] LustreError: 196166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778928091 with bad export cookie 13686733767070244384 [ 5369.283478] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5369.286221] LustreError: 196166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5369.439951] Lustre: server umount lustre-MDT0001 complete [ 5373.434691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5375.413351] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5379.139325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5380.936935] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5383.279988] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5384.948974] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5388.332642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5391.174188] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5393.583491] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5395.440821] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5399.658262] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 5401.337099] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5401.429150] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5404.130233] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5404.134692] LustreError: Skipped 1 previous similar message [ 5408.368824] Lustre: server umount lustre-OST0000 complete [ 5410.761457] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5412.438870] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5436.930499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5437.102630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5437.105989] Lustre: Skipped 5 previous similar messages [ 5439.796615] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5442.141295] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5442.971775] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5444.262706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5444.273477] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5444.275385] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5444.286285] Lustre: Skipped 1 previous similar message [ 5450.798734] Lustre: server umount lustre-OST0000 complete [ 5458.796491] Lustre: server umount lustre-MDT0000 complete [ 5459.936727] LustreError: 199211:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5459.946171] LustreError: 199211:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 5460.532349] LustreError: 199175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778928183 with bad export cookie 13686733767070245266 [ 5460.534838] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5460.537473] LustreError: 199175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5460.680777] Lustre: server umount lustre-MDT0001 complete [ 5464.922631] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 06:43:06 (1778928186) [ 5468.481117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5470.916658] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5475.324193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5477.294571] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5479.629480] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5481.048412] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5483.995132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5486.593757] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5488.969451] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5490.697557] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5493.881848] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 5495.836178] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5495.934811] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5498.919491] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5499.004607] Lustre: lustre-OST0001: new disk, initializing [ 5499.007578] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5499.011639] Lustre: 207668:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5501.547744] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5505.061135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5505.067390] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5505.130303] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5505.491626] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5507.568440] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5507.695811] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5509.610539] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5509.710448] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5515.234936] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5515.235612] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5515.239201] LustreError: Skipped 3 previous similar messages [ 5515.242954] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5515.245732] Lustre: Skipped 10 previous similar messages [ 5515.247828] Lustre: Skipped 8 previous similar messages [ 5516.624224] Lustre: server umount lustre-OST0001 complete [ 5519.133558] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5519.228656] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5520.953824] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5521.055443] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5545.276424] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5547.528651] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5549.629357] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5550.306742] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5550.573274] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5550.582270] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5550.583611] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5550.591785] Lustre: Skipped 1 previous similar message [ 5553.221356] Lustre: server umount lustre-OST0001 complete [ 5571.694696] Lustre: server umount lustre-OST0000 complete [ 5579.540937] Lustre: server umount lustre-MDT0000 complete [ 5580.987303] LustreError: 204636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778928303 with bad export cookie 13686733767070246204 [ 5580.990060] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5580.993515] LustreError: 204636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5587.204778] Lustre: server umount lustre-MDT0001 complete [ 5596.583503] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 06:45:18 (1778928318) [ 5599.232333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5599.387644] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5599.402681] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5599.492791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5599.496617] Lustre: Skipped 5 previous similar messages [ 5601.124318] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5604.551308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5604.613647] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5604.628208] Lustre: 212608:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5606.374977] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5608.453593] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5609.889709] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5612.505692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5612.599957] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5615.000563] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5617.297196] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5621.781674] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5621.831574] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5624.152368] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5626.423440] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5627.112690] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5628.743619] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5631.909815] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 5633.512992] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5633.611061] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5635.155184] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90f0c41ef000.ost_server_uuid 50 [ 5635.772943] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90f0c41ef000.ost_server_uuid in FULL state after 0 sec [ 5636.168254] Lustre: Permanently deactivating lustre-OST0001 [ 5636.175430] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5637.119741] Lustre: Permanently reactivating lustre-OST0001 [ 5644.490714] Lustre: server umount lustre-OST0001 complete [ 5646.821252] LustreError: 214225:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5646.828726] LustreError: 214225:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 26 previous similar messages [ 5652.121390] Lustre: server umount lustre-OST0000 complete [ 5656.033200] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5656.036796] LustreError: Skipped 5 previous similar messages [ 5659.680075] Lustre: server umount lustre-MDT0000 complete [ 5661.174384] LustreError: 212643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778928383 with bad export cookie 13686733767070247485 [ 5661.176824] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5661.185237] LustreError: 212643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5661.333612] Lustre: server umount lustre-MDT0001 complete [ 5671.602595] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 06:46:33 (1778928393) [ 5674.771741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5674.956027] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5674.986303] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5674.997779] Lustre: Skipped 5 previous similar messages [ 5676.970916] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5680.723474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5680.797536] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5680.809397] Lustre: 218730:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5682.604524] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5684.820424] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5686.392682] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5689.027438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5689.124942] Lustre: Permanently deactivating lustre-OST0000 [ 5689.131099] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5691.379447] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5691.639642] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5693.287018] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5695.891337] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5698.279946] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5698.752889] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 5700.608975] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5701.390607] Lustre: Permanently reactivating lustre-OST0000 [ 5708.806671] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5708.810793] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.815969] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5708.830599] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 5708.838737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5720.002820] Lustre: server umount lustre-OST0001 complete [ 5727.443870] Lustre: server umount lustre-OST0000 complete [ 5735.011866] Lustre: server umount lustre-MDT0000 complete [ 5736.382363] LustreError: 218725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778928459 with bad export cookie 13686733767070249123 [ 5736.386246] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5736.387704] LustreError: 218725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5736.508546] Lustre: server umount lustre-MDT0001 complete [ 5740.702253] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 5741.972413] Key type lgssc unregistered [ 5742.127311] LNet: 223132:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5742.131418] LNetError: 223132:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5743.142722] LNet: Removed LNI 192.168.203.116@tcp [ 5743.487412] Key type .llcrypt unregistered [ 5743.488929] Key type ._llcrypt unregistered [ 5752.956616] Key type ._llcrypt registered [ 5752.958181] Key type .llcrypt registered [ 5753.004152] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 5753.471118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5753.481924] alg: No test for adler32 (adler32-zlib) [ 5754.364790] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 5754.469990] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 5756.064132] Key type lgssc registered [ 5756.529963] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5760.382625] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5760.399708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5761.649307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5763.284694] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5766.901056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5766.968372] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5766.987819] Lustre: 224547:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5766.994825] Lustre: Permanently deactivating lustre-MDT0001 [ 5767.002599] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5767.155862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5768.780423] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5771.057619] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5771.061320] LustreError: 224590:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5771.066213] LustreError: 224590:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5771.071144] LustreError: 224590:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 1, failed: rc = -108 [ 5771.167389] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5772.742402] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5775.444194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5775.646532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5776.675109] LustreError: 226174:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5776.680194] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 5776.683261] LustreError: 226174:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5778.206807] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5780.428323] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5781.986465] LustreError: 226174:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5783.310616] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5783.422783] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5785.799683] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5786.473844] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 5786.476052] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 5786.476620] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 5788.136495] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5790.866378] Lustre: Permanently reactivating lustre-MDT0001 [ 5790.873861] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 5790.877212] Lustre: Skipped 2 previous similar messages [ 5795.435563] Lustre: lustre-MDT0001: Client 58e3d062-5246-4508-8eb6-6ea98b09b80b (at 192.168.203.16@tcp) reconnecting [ 5797.761921] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5798.481573] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5800.546116] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5800.643501] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5802.259781] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5802.353487] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5803.429116] Lustre: Permanently deactivating lustre-MDT0001 [ 5803.437405] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5803.441085] Lustre: Skipped 2 previous similar messages [ 5809.163968] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5812.194483] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5812.196726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5812.200252] Lustre: Skipped 3 previous similar messages [ 5812.203755] Lustre: Skipped 2 previous similar messages [ 5817.313481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5817.315855] Lustre: Skipped 4 previous similar messages [ 5817.762256] Lustre: server umount lustre-MDT0000 complete [ 5819.487462] LustreError: 224540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778928542 with bad export cookie 13165737097787530275 [ 5819.489135] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5819.493476] LustreError: 224540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5819.639257] Lustre: server umount lustre-MDT0001 complete [ 5827.451567] Lustre: server umount lustre-OST0000 complete [ 5831.359240] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 06:49:13 (1778928553) [ 5835.284527] Lustre: server umount lustre-OST0001 complete [ 5837.589449] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 5841.091785] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 5855.334869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5855.445474] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5855.450844] Lustre: Skipped 2 previous similar messages [ 5855.465224] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5855.506144] Lustre: lustre-MDT0000: new disk, initializing [ 5855.538860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5855.548852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5857.052867] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5861.438181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5861.484693] Lustre: 232236:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5861.512885] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5861.515865] Lustre: Skipped 1 previous similar message [ 5861.554809] Lustre: lustre-MDT0001: new disk, initializing [ 5861.603803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5861.608147] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5863.018391] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5866.462979] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5867.926378] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5870.702086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5870.850084] Lustre: lustre-OST0000: new disk, initializing [ 5870.852095] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5870.855326] Lustre: 233977:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5870.888491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5870.891617] Lustre: Skipped 1 previous similar message [ 5873.236500] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5876.696543] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5881.341761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5881.346664] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5881.378260] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5882.642303] LustreError: 232244:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5887.190508] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5887.259305] Lustre: lustre-OST0001: new disk, initializing [ 5887.261390] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5887.265037] Lustre: 234979:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5887.300384] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5889.353681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5889.358053] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5889.389957] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5889.506271] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5892.738343] LustreError: 232244:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5892.748849] LustreError: 233366:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5893.013112] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5893.715057] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5902.840132] LustreError: 233366:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5904.866640] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5904.866758] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5904.871517] LustreError: Skipped 1 previous similar message [ 5904.881145] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5904.885261] Lustre: Skipped 1 previous similar message [ 5909.640841] Lustre: server umount lustre-OST0001 complete [ 5909.989404] LustreError: 233960:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5909.998851] LustreError: 233960:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5915.107869] LustreError: 234583:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5915.118022] LustreError: 234583:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5920.230607] LustreError: 233961:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5920.239177] LustreError: 233961:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5922.273337] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5922.273377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5922.278575] LustreError: Skipped 1 previous similar message [ 5922.289678] Lustre: Skipped 1 previous similar message [ 5922.291983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5922.300860] Lustre: Skipped 2 previous similar messages [ 5928.161255] Lustre: server umount lustre-OST0000 complete [ 5930.465769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5930.466480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5930.472125] Lustre: Skipped 2 previous similar messages [ 5930.475602] Lustre: Skipped 2 previous similar messages [ 5935.909903] Lustre: server umount lustre-MDT0000 complete [ 5937.525142] LustreError: 235723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778928660 with bad export cookie 13165737097787533285 [ 5937.526743] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5937.535407] LustreError: 235723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5943.755864] Lustre: server umount lustre-MDT0001 complete [ 5948.253658] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 5949.483649] Key type lgssc unregistered [ 5949.628508] LNet: 236882:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5949.632032] LNetError: 236882:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5949.639327] LNet: Removed LNI 192.168.203.116@tcp [ 5949.959381] Key type .llcrypt unregistered [ 5949.961062] Key type ._llcrypt unregistered [ 5966.906522] Key type ._llcrypt registered [ 5966.908098] Key type .llcrypt registered [ 5966.955394] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 5967.418139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5967.448771] alg: No test for adler32 (adler32-zlib) [ 5968.318842] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 5968.415724] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 5970.016164] Key type lgssc registered [ 5970.472471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5973.758974] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5973.770361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5974.905551] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5974.919208] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5975.001279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5976.634592] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5980.323846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5980.375615] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5980.389169] Lustre: 239050:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5980.500637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5982.129324] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5984.430523] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5985.825988] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5988.484318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5988.571195] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5988.651626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5990.685318] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5991.526858] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 5992.811170] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6037.472742] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6037.476057] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6037.482050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6037.987272] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6037.987474] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6037.993990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6039.457690] Lustre: server umount lustre-OST0000 complete [ 6041.004687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6045.488452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6045.610129] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6047.653903] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6049.706764] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6057.892482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.897666] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.897688] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6057.905073] Lustre: Skipped 1 previous similar message [ 6083.553489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6083.554860] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6083.556102] LustreError: Skipped 1 previous similar message [ 6083.560813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6083.563607] Lustre: Skipped 1 previous similar message [ 6088.674206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6088.676772] Lustre: Skipped 1 previous similar message [ 6089.566797] Lustre: server umount lustre-OST0000 complete [ 6093.280991] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6093.283781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6093.289253] Lustre: Skipped 1 previous similar message [ 6093.291468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6097.113328] Lustre: server umount lustre-MDT0000 complete [ 6098.443380] LustreError: 239042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778928821 with bad export cookie 13007191141179257580 [ 6098.444172] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6098.448882] LustreError: 239042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6098.570703] Lustre: server umount lustre-MDT0001 complete [ 6102.370529] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 6103.471560] Key type lgssc unregistered [ 6103.607481] LNet: 243992:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6103.611450] LNetError: 243992:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6103.621653] LNet: Removed LNI 192.168.203.116@tcp [ 6103.923302] Key type .llcrypt unregistered [ 6103.925073] Key type ._llcrypt unregistered [ 6113.979945] Key type ._llcrypt registered [ 6113.981476] Key type .llcrypt registered [ 6114.024210] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 6114.453298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6114.508688] alg: No test for adler32 (adler32-zlib) [ 6115.368294] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 6115.455823] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 6117.048137] Key type lgssc registered [ 6117.455403] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6120.229811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6120.236617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6121.406048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6122.797668] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6126.059249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6126.211892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6127.664674] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6129.625206] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6130.931669] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6133.257994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6133.390897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6135.639092] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6137.704957] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6146.724895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 6156.257912] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6156.260419] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6156.266414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6157.282919] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6157.283027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6157.283357] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6157.294181] Lustre: Skipped 1 previous similar message [ 6162.323135] Lustre: server umount lustre-OST0000 complete [ 6167.521773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6167.521881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6167.522517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6167.534335] Lustre: Skipped 1 previous similar message [ 6172.641739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6172.647744] Lustre: Skipped 2 previous similar messages [ 6177.760978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6177.763695] Lustre: Skipped 1 previous similar message [ 6178.272103] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6178.333157] Lustre: server umount lustre-MDT0000 complete [ 6179.689312] LustreError: 245348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778928902 with bad export cookie 3814672841708300812 [ 6179.692522] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6179.694466] LustreError: 245348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6179.810755] Lustre: server umount lustre-MDT0001 complete [ 6183.553977] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 6184.664315] Key type lgssc unregistered [ 6184.800596] LNet: 249334:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6184.803566] LNetError: 249334:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6184.812309] LNet: Removed LNI 192.168.203.116@tcp [ 6185.110110] Key type .llcrypt unregistered [ 6185.111902] Key type ._llcrypt unregistered [ 6192.519767] Key type ._llcrypt registered [ 6192.521226] Key type .llcrypt registered [ 6192.564229] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 6192.972484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6193.027480] alg: No test for adler32 (adler32-zlib) [ 6193.899753] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 6193.989380] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 6195.576158] Key type lgssc registered [ 6195.984259] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6199.054258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6199.062100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6200.238778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6201.758706] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6205.080242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6205.253554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6206.783887] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6208.829887] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6210.084603] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6212.637274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6212.769945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6214.949710] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6217.147812] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6225.068645] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6235.617906] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6235.620657] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6235.620810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6235.623277] Lustre: Skipped 1 previous similar message [ 6237.331576] Lustre: server umount lustre-OST0000 complete [ 6240.738221] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6240.739123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6240.742885] Lustre: Skipped 1 previous similar message [ 6240.744724] Lustre: Skipped 1 previous similar message [ 6244.825100] Lustre: server umount lustre-MDT0000 complete [ 6245.857672] LustreError: 250518:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6245.866932] LustreError: 250518:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 6246.192376] LustreError: 250498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778928968 with bad export cookie 16466142969112007376 [ 6246.196458] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6246.196870] LustreError: 250498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6246.331981] Lustre: server umount lustre-MDT0001 complete [ 6250.047880] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 6251.198743] Key type lgssc unregistered [ 6251.332725] LNet: 253861:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6251.335854] LNetError: 253861:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6251.343586] LNet: Removed LNI 192.168.203.116@tcp [ 6251.678600] Key type .llcrypt unregistered [ 6251.679626] Key type ._llcrypt unregistered [ 6261.141516] Key type ._llcrypt registered [ 6261.142775] Key type .llcrypt registered [ 6261.188973] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 6261.522845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6261.567089] alg: No test for adler32 (adler32-zlib) [ 6262.426696] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 6262.517628] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 6264.096155] Key type lgssc registered [ 6264.511102] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6267.279484] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6267.287094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6268.461744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6269.753731] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6272.687345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6272.826527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6274.224727] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6276.270779] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6277.604235] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6280.017799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6280.138306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6282.157793] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6284.022580] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6293.480080] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6303.717693] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6303.717805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6303.726574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6303.729563] Lustre: Skipped 1 previous similar message [ 6308.632385] Lustre: server umount lustre-OST0000 complete [ 6313.953858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6313.954900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6313.955496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6313.956887] LustreError: Skipped 1 previous similar message [ 6313.961600] Lustre: Skipped 2 previous similar messages [ 6316.189831] Lustre: server umount lustre-MDT0000 complete [ 6317.585220] LustreError: 255218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778929040 with bad export cookie 5977164851987991951 [ 6317.588806] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6317.588955] LustreError: 255218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6317.725053] Lustre: server umount lustre-MDT0001 complete [ 6321.429309] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 6322.522351] Key type lgssc unregistered [ 6322.659418] LNet: 259203:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6322.663867] LNetError: 259203:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6322.671332] LNet: Removed LNI 192.168.203.116@tcp [ 6322.985890] Key type .llcrypt unregistered [ 6322.987590] Key type ._llcrypt unregistered [ 6330.526274] Key type ._llcrypt registered [ 6330.527431] Key type .llcrypt registered [ 6330.578628] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 6330.966959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6331.022507] alg: No test for adler32 (adler32-zlib) [ 6331.881826] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 6331.960883] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 6333.544150] Key type lgssc registered [ 6333.914740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6336.566653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6336.574537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6337.757322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6339.096728] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6342.069113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6342.208722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6343.649174] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6345.545758] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6346.777827] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6349.149485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6349.292188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6351.163919] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6352.972922] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6362.602461] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 6367.717035] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6367.717835] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6367.718565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6367.726078] Lustre: Skipped 1 previous similar message [ 6369.292509] Lustre: server umount lustre-OST0000 complete [ 6372.567649] Lustre: server umount lustre-MDT0000 complete [ 6372.834654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6372.835767] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6372.837908] LustreError: 260404:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6372.841059] Lustre: Skipped 1 previous similar message [ 6372.847693] LustreError: 260404:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 6373.814335] LustreError: 262738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778929096 with bad export cookie 2947433625433729731 [ 6373.817319] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6373.819073] LustreError: 262738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6373.944063] Lustre: server umount lustre-MDT0001 complete [ 6377.502765] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 6378.601483] Key type lgssc unregistered [ 6378.729536] LNet: 263760:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6378.733186] LNetError: 263760:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6378.746306] LNet: Removed LNI 192.168.203.116@tcp [ 6379.042883] Key type .llcrypt unregistered [ 6379.044249] Key type ._llcrypt unregistered [ 6382.683775] Key type ._llcrypt registered [ 6382.685288] Key type .llcrypt registered [ 6382.721530] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 6388.298299] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 6394.698176] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 6395.114466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6395.161942] alg: No test for adler32 (adler32-zlib) [ 6396.012799] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 6396.099169] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 6397.680158] Key type lgssc registered [ 6398.079814] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6409.441582] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6409.447723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6410.532798] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6410.544604] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6410.583159] Lustre: lustre-MDT0000: new disk, initializing [ 6410.606546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6410.612695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6411.838377] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6415.964662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6416.004027] Lustre: 267232:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6416.027694] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6416.030098] Lustre: Skipped 1 previous similar message [ 6416.065967] Lustre: lustre-MDT0001: new disk, initializing [ 6416.096568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6416.107332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6416.110818] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6417.448980] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6420.663681] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6421.900192] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6424.144422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6424.245635] Lustre: lustre-OST0000: new disk, initializing [ 6424.248132] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6424.252356] Lustre: 268973:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6424.282660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6426.304965] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6429.469717] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6430.864040] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6431.735393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6431.740089] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6431.765755] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6431.965407] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 6433.313010] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6433.392619] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6436.836843] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6436.837804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6436.845160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6436.847505] Lustre: Skipped 1 previous similar message [ 6440.146455] Lustre: server umount lustre-OST0000 complete [ 6441.953289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6441.954172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6441.954200] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6441.954205] LustreError: Skipped 1 previous similar message [ 6441.957938] Lustre: Skipped 2 previous similar messages [ 6447.073598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6447.076157] Lustre: Skipped 2 previous similar messages [ 6447.575635] Lustre: server umount lustre-MDT0000 complete [ 6448.824978] LustreError: 267223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778929171 with bad export cookie 1875217670809149204 [ 6448.827450] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6448.828957] LustreError: 267223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6448.950147] Lustre: server umount lustre-MDT0001 complete [ 6451.696857] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 06:59:33 (1778929173) [ 6453.868772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6454.032111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6455.242743] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6458.006261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6458.141156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6459.468082] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6461.321881] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6462.507660] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6464.741244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6466.690413] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6468.474263] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6488.544746] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6488.547951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6488.552286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6490.595934] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6490.596122] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6493.147405] Lustre: server umount lustre-OST0000 complete [ 6495.713867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6495.714775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6495.717877] Lustre: Skipped 1 previous similar message [ 6495.721449] Lustre: Skipped 3 previous similar messages [ 6500.695159] Lustre: server umount lustre-MDT0000 complete [ 6500.832757] LustreError: 271241:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6500.840982] LustreError: 271241:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 6501.964527] LustreError: 271222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778929224 with bad export cookie 1875217670809150072 [ 6501.966973] LustreError: 271961:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 6501.967994] LustreError: 271222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6501.969071] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6502.088319] Lustre: server umount lustre-MDT0001 complete [ 6505.755866] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 6506.871360] Key type lgssc unregistered [ 6507.009424] LNet: 274440:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6507.012667] LNetError: 274440:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6507.020436] LNet: Removed LNI 192.168.203.116@tcp [ 6507.336632] Key type .llcrypt unregistered [ 6507.338356] Key type ._llcrypt unregistered [ 6516.035413] Key type ._llcrypt registered [ 6516.036669] Key type .llcrypt registered [ 6516.076736] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 6522.012977] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 6522.447815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6522.458594] alg: No test for adler32 (adler32-zlib) [ 6523.314942] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 6523.402568] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 6524.984134] Key type lgssc registered [ 6525.335673] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6539.539522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6539.545182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6540.624807] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6540.635866] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6540.669535] Lustre: lustre-MDT0000: new disk, initializing [ 6540.694259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6540.700796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6541.972914] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6545.893810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6545.931222] Lustre: 278211:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6545.954258] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6545.955897] Lustre: Skipped 1 previous similar message [ 6545.993975] Lustre: lustre-MDT0001: new disk, initializing [ 6546.026922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6546.040553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6546.043906] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6547.371243] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6550.342355] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6551.426803] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6553.605127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6553.706626] Lustre: lustre-OST2710: new disk, initializing [ 6553.708371] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 6553.711233] Lustre: 279953:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6553.740636] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 6555.524083] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6558.503426] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 6558.720647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 6558.725870] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 6558.762088] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 6560.750827] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6560.812539] Lustre: lustre-OST03e8: new disk, initializing [ 6560.814359] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 6560.817180] Lustre: 280904:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6560.851642] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 6562.627080] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6564.448716] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 6566.913188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 6566.917978] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 6566.940212] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 6567.050156] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 6567.121102] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6568.459804] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 6568.546578] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6569.850969] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 6569.932980] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6571.265036] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 6571.336344] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6575.011079] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6575.012696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6575.014879] Lustre: Skipped 2 previous similar messages [ 6575.017765] Lustre: Skipped 1 previous similar message [ 6578.529051] Lustre: server umount lustre-MDT0000 complete [ 6579.747442] LustreError: 278206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778929302 with bad export cookie 8300143318817235112 [ 6579.750239] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6579.752619] LustreError: 278206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6579.855857] Lustre: server umount lustre-MDT0001 complete [ 6591.364557] Lustre: server umount lustre-OST2710 complete [ 6602.841706] Lustre: server umount lustre-OST03e8 complete [ 6608.624279] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 6611.864093] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 6623.359874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6623.440594] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6623.451735] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6623.486628] Lustre: lustre-MDT0000: new disk, initializing [ 6623.509791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6623.516199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6624.669093] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6628.595524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6628.634308] Lustre: 285930:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6628.729390] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6629.970204] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6632.850996] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6633.976564] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6636.144470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6636.240156] Lustre: lustre-OST0000: new disk, initializing [ 6636.241732] Lustre: Skipped 1 previous similar message [ 6636.243729] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6636.246406] Lustre: Skipped 2 previous similar messages [ 6636.248882] Lustre: 287670:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6636.274598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6636.276913] Lustre: Skipped 1 previous similar message [ 6637.811987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6637.814618] Lustre: Skipped 1 previous similar message [ 6637.816514] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6637.845830] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6638.097886] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6641.124133] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6642.427290] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6642.494508] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6643.851920] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6643.925375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6648.289744] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6648.291263] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6648.293083] LustreError: Skipped 1 previous similar message [ 6648.293629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6648.298090] Lustre: Skipped 2 previous similar messages [ 6650.702888] Lustre: server umount lustre-OST0000 complete [ 6653.409660] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6653.410177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6653.414714] Lustre: Skipped 1 previous similar message [ 6653.416668] Lustre: Skipped 1 previous similar message [ 6658.529050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6658.530823] Lustre: Skipped 1 previous similar message [ 6663.648994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6663.651818] Lustre: Skipped 1 previous similar message [ 6666.208107] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6666.273871] Lustre: server umount lustre-MDT0000 complete [ 6667.585556] LustreError: 285920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778929390 with bad export cookie 8300143318817236771 [ 6667.588651] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6667.590882] LustreError: 285920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6667.708707] Lustre: server umount lustre-MDT0001 complete [ 6670.571909] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 07:03:12 (1778929392) [ 6671.125605] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 6671.744618] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 07:03:13 (1778929393) [ 6672.737532] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 6681.700020] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6681.783744] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6681.790602] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6681.844894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6683.051708] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6685.824184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6685.861549] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6685.872967] Lustre: 291356:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6687.255200] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6703.138658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6703.205268] Lustre: Denying initial registration attempt from nid 192.168.203.116@tcp, specified as failover [ 6703.208440] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 6703.211668] LustreError: 292767:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 6703.214220] LustreError: 292767:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 6703.216115] LustreError: 292767:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 6703.236330] Lustre: server umount lustre-OST0000 complete [ 6703.237618] LustreError: 292767:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 6706.145369] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6706.145850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6706.149655] Lustre: Skipped 1 previous similar message [ 6706.152807] Lustre: Skipped 1 previous similar message [ 6711.315864] Lustre: server umount lustre-MDT0000 complete [ 6712.569213] LustreError: 291346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778929435 with bad export cookie 8300143318817237639 [ 6712.572780] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6712.573206] LustreError: 291346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6712.675898] Lustre: server umount lustre-MDT0001 complete [ 6716.042791] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 6716.998253] Key type lgssc unregistered [ 6717.123325] LNet: 293875:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6717.126554] LNetError: 293875:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6717.137342] LNet: Removed LNI 192.168.203.116@tcp [ 6717.438033] Key type .llcrypt unregistered [ 6717.439179] Key type ._llcrypt unregistered [ 6721.023619] Key type ._llcrypt registered [ 6721.024680] Key type .llcrypt registered [ 6721.062359] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 6721.616381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6721.639105] alg: No test for adler32 (adler32-zlib) [ 6722.497844] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 6722.581569] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 6724.160150] Key type lgssc registered [ 6724.520990] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6747.649853] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 6750.701827] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6750.708708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6751.794587] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6751.801670] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6751.856819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6753.136294] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6756.079818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6756.114217] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6756.124904] Lustre: 296579:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6756.215229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6757.753197] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6759.527525] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6760.659938] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6775.907325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6775.986486] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6776.048154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6777.990519] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6779.815994] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6780.544507] Lustre: server umount lustre-OST0000 complete [ 6781.921286] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6781.925348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6781.931328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6787.041509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6787.042215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6792.161249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6792.163944] Lustre: Skipped 2 previous similar messages [ 6796.256103] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6796.310076] Lustre: server umount lustre-MDT0000 complete [ 6797.491022] LustreError: 297303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778929520 with bad export cookie 3092245125363630776 [ 6797.493456] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6797.496799] LustreError: 297303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6797.611991] Lustre: server umount lustre-MDT0001 complete [ 6800.987912] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 6801.958500] Key type lgssc unregistered [ 6802.086330] LNet: 299786:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6802.089287] LNetError: 299786:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6802.097279] LNet: Removed LNI 192.168.203.116@tcp [ 6802.367392] Key type .llcrypt unregistered [ 6802.368540] Key type ._llcrypt unregistered [ 6810.916230] Key type ._llcrypt registered [ 6810.917786] Key type .llcrypt registered [ 6810.955248] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 6811.253387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6811.308861] alg: No test for adler32 (adler32-zlib) [ 6812.154592] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 6812.233505] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 6813.816209] Key type lgssc registered [ 6814.148196] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6816.585154] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6816.591386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6817.741966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6818.935960] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6821.585990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6821.702580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6822.987269] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6824.719559] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6825.912437] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6830.114938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6830.216402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6831.922402] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6833.531605] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6847.456718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6847.459128] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6847.463307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6847.970473] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6847.971530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6847.974498] Lustre: Skipped 1 previous similar message [ 6849.120791] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.16@tcp (stopping) [ 6849.123092] Lustre: Skipped 1 previous similar message [ 6852.118365] Lustre: server umount lustre-MDT0000 complete [ 6853.089248] LustreError: 301884:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6853.094846] LustreError: 301884:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 6853.255092] LustreError: 303423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778929575 with bad export cookie 1691214386304799234 [ 6853.257980] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6853.259344] LustreError: 303423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6854.233492] LustreError: 301186:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6854.240823] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.16@tcp (stopping) [ 6858.208841] LustreError: 301156:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6858.208995] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6859.353906] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.16@tcp (stopping) [ 6859.356764] Lustre: Skipped 1 previous similar message [ 6859.436387] Lustre: server umount lustre-MDT0001 complete [ 6861.446765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6863.999085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6878.689023] LustreError: 300472:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000030d5074d x1865342815067136/t0(0) o250->MGC192.168.203.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6878.786350] LustreError: 302764:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6878.792891] LustreError: 302764:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 6878.807179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6878.825423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 6879.970552] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6882.599033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6883.946789] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6885.683307] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6888.868318] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6891.271540] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 6892.309618] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6892.785823] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6894.049555] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6894.052087] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6894.054089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6894.055599] Lustre: Skipped 1 previous similar message [ 6908.384084] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6908.444218] Lustre: server umount lustre-OST0000 complete [ 6913.504795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6913.507040] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6913.511293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6913.512990] Lustre: Skipped 5 previous similar messages [ 6915.731954] Lustre: server umount lustre-MDT0000 complete [ 6916.851971] LustreError: 301141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778929639 with bad export cookie 1691214386304806388 [ 6916.854233] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6916.855283] LustreError: 301141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6916.955780] Lustre: server umount lustre-MDT0001 complete [ 6920.042560] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 6920.910304] Key type lgssc unregistered [ 6921.022298] LNet: 306862:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6921.024541] LNetError: 306862:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6921.036300] LNet: Removed LNI 192.168.203.116@tcp [ 6921.304108] Key type .llcrypt unregistered [ 6921.305728] Key type ._llcrypt unregistered [ 6929.723067] Key type ._llcrypt registered [ 6929.724569] Key type .llcrypt registered [ 6929.761836] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 6930.213592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6930.254537] alg: No test for adler32 (adler32-zlib) [ 6931.122825] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 6931.212760] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 6932.808140] Key type lgssc registered [ 6933.211565] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6935.901236] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6935.907483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6937.073945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6938.370398] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6941.111819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6941.228743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6942.468802] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6944.174822] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6945.312315] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6946.251495] LustreError: 308268:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6946.253335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6946.253860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6946.255346] LustreError: 308268:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6946.259243] Lustre: Skipped 1 previous similar message [ 6946.267647] LustreError: 308268:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 6950.369242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6950.371549] Lustre: Skipped 2 previous similar messages [ 6952.404043] Lustre: server umount lustre-MDT0000 complete [ 6953.635186] LustreError: 308218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778929676 with bad export cookie 13743394223459260669 [ 6953.638232] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6953.639598] LustreError: 308218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6953.754558] Lustre: server umount lustre-MDT0001 complete [ 6958.670871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6958.776938] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6958.787966] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6958.860691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6960.102021] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6962.819772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6962.849943] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6962.860396] Lustre: 310594:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6962.954092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6964.186811] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6965.913465] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6967.035936] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6971.490738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6971.557129] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 6971.561157] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 6971.565578] LustreError: 312249:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 6971.568520] LustreError: 312249:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 6971.570305] LustreError: 312249:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 6971.595781] Lustre: server umount lustre-OST0000 complete [ 6971.597428] LustreError: 312249:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6976.290697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6976.352712] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6976.409532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6978.139257] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6979.821488] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6982.373689] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6982.405919] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6982.430036] Lustre: lustre-OST0001: new disk, initializing [ 6982.432403] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6982.434817] Lustre: 313665:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6984.323973] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6986.058659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6986.063272] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6986.070327] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6986.083518] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 6986.104220] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6991.333446] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6991.335877] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6991.335958] LustreError: Skipped 1 previous similar message [ 6991.342159] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6991.344551] Lustre: Skipped 1 previous similar message [ 6993.153573] Lustre: server umount lustre-OST0001 complete [ 6996.450471] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6996.450773] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6996.451418] LustreError: 312715:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6996.451430] LustreError: 312715:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6996.453559] LustreError: Skipped 1 previous similar message [ 6996.457856] Lustre: Skipped 1 previous similar message [ 6996.463867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6996.472363] Lustre: Skipped 1 previous similar message [ 7000.534862] Lustre: server umount lustre-OST0000 complete [ 7004.128692] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7004.131394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7004.135810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7007.966883] Lustre: server umount lustre-MDT0000 complete [ 7009.169141] LustreError: 310585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778929731 with bad export cookie 13743394223459261187 [ 7009.171774] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7009.172860] LustreError: 310585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7009.289185] Lustre: server umount lustre-MDT0001 complete [ 7012.565178] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 7013.547522] Key type lgssc unregistered [ 7013.684407] LNet: 315497:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7013.687555] LNetError: 315497:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7013.696385] LNet: Removed LNI 192.168.203.116@tcp [ 7013.988381] Key type .llcrypt unregistered [ 7013.989473] Key type ._llcrypt unregistered [ 7044.287658] Key type ._llcrypt registered [ 7044.288821] Key type .llcrypt registered [ 7044.327807] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 7050.385475] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7050.699713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7050.743035] alg: No test for adler32 (adler32-zlib) [ 7051.588565] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 7051.670420] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 7053.248142] Key type lgssc registered [ 7053.589603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7064.914240] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7064.922073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7066.014172] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7066.024990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7066.062713] Lustre: lustre-MDT0000: new disk, initializing [ 7066.084341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7066.090736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7067.336096] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7071.327570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7071.362608] Lustre: 320363:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7071.386183] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7071.389031] Lustre: Skipped 1 previous similar message [ 7071.426409] Lustre: lustre-MDT0001: new disk, initializing [ 7071.454953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7071.466068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7071.468906] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7072.654062] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7075.589649] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7076.729753] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7078.933854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7079.024454] Lustre: lustre-OST0000: new disk, initializing [ 7079.025820] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7079.027838] Lustre: 322104:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7079.054691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7080.834344] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7083.784655] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7085.151560] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7085.557218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7085.560326] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7085.587014] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7086.238305] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 7087.392979] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7087.454490] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7090.657869] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7090.658654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7090.660346] LustreError: Skipped 1 previous similar message [ 7090.660554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7090.664035] Lustre: Skipped 1 previous similar message [ 7094.158881] Lustre: server umount lustre-OST0000 complete [ 7095.777654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7095.778467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7095.783134] Lustre: Skipped 1 previous similar message [ 7095.785424] Lustre: Skipped 1 previous similar message [ 7100.897711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7100.900295] Lustre: Skipped 2 previous similar messages [ 7101.526917] Lustre: server umount lustre-MDT0000 complete [ 7102.736934] LustreError: 320357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778929825 with bad export cookie 8800605471121416104 [ 7102.739972] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7102.740590] LustreError: 320357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7102.851068] Lustre: server umount lustre-MDT0001 complete [ 7105.577037] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 07:10:27 (1778929827) [ 7108.017538] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 07:10:30 (1778929830) [ 7110.708631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7110.866253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7112.139172] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7114.908115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7115.026978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7116.296294] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7117.960701] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7119.081437] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7121.261369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7123.170206] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7124.949122] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7130.260094] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 7130.791059] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7131.331116] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7131.862691] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 7132.465805] Lustre: Failing over lustre-MDT0000 [ 7132.633435] Lustre: server umount lustre-MDT0000 complete [ 7134.689347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7134.691011] LustreError: 326327:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7134.693288] Lustre: Skipped 2 previous similar messages [ 7134.699297] LustreError: 326327:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 7137.377680] LustreError: 326327:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7139.810215] LustreError: 325427:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7139.815962] LustreError: 325427:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 7142.490373] LustreError: 324715:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7145.440893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7145.478101] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7145.561197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7145.563848] Lustre: Skipped 1 previous similar message [ 7145.591725] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7146.793248] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7147.610127] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7150.568798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7150.577094] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 7150.592370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7151.681118] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7152.251147] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7154.158589] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 7154.805051] Lustre: Failing over lustre-MDT0000 [ 7154.921289] Lustre: server umount lustre-MDT0000 complete [ 7155.681510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7155.682047] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7155.682945] LustreError: 324719:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7155.682952] LustreError: 324719:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 7155.698057] Lustre: Skipped 2 previous similar messages [ 7165.922878] LustreError: 324720:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7165.928462] LustreError: 324720:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 9 previous similar messages [ 7167.778410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7167.825599] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7167.919800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7167.939764] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7168.089440] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7169.144643] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7173.093636] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7173.095759] Lustre: Skipped 2 previous similar messages [ 7173.104644] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 7173.120950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7174.280095] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7174.786516] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7178.210663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7178.210919] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7178.211509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7178.218927] Lustre: Skipped 2 previous similar messages [ 7183.063449] Lustre: server umount lustre-MDT0000 complete [ 7183.332506] LustreError: 324719:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7183.338209] LustreError: 324719:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 7184.285189] LustreError: 326331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778929906 with bad export cookie 8800605471121418736 [ 7184.288423] LustreError: 326331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7184.288513] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7184.404120] Lustre: server umount lustre-MDT0001 complete [ 7195.928318] Lustre: server umount lustre-OST0000 complete [ 7200.744781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7200.902410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7202.098438] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7204.886636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7206.291631] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7208.010183] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7209.218264] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7211.430412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7213.380593] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7214.635675] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7215.117669] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7219.683468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7219.684396] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7219.686460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7219.686465] Lustre: Skipped 2 previous similar messages [ 7219.688543] Lustre: Skipped 1 previous similar message [ 7219.697575] LustreError: Skipped 1 previous similar message [ 7222.353202] Lustre: server umount lustre-OST0000 complete [ 7229.722927] Lustre: server umount lustre-MDT0000 complete [ 7229.921335] LustreError: 330479:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7229.926767] LustreError: 330479:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 7230.900048] LustreError: 332096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778929953 with bad export cookie 8800605471121419492 [ 7230.902777] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7230.903215] LustreError: 332096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7231.016471] Lustre: server umount lustre-MDT0001 complete [ 7234.386942] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 7235.399828] Key type lgssc unregistered [ 7235.530358] LNet: 333654:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7235.532823] LNetError: 333654:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7235.541260] LNet: Removed LNI 192.168.203.116@tcp [ 7235.831245] Key type .llcrypt unregistered [ 7235.832691] Key type ._llcrypt unregistered [ 7245.392953] Key type ._llcrypt registered [ 7245.394167] Key type .llcrypt registered [ 7245.435630] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7245.969595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7246.014255] alg: No test for adler32 (adler32-zlib) [ 7246.895753] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 7246.986495] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 7248.568182] Key type lgssc registered [ 7248.946053] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7251.443323] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7251.448978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7252.606033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7253.783522] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7256.447225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7256.562635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7257.780399] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7259.561958] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7260.723065] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7262.832301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7262.934728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7264.607358] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7266.294216] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7278.313492] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7287.265647] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7287.268626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7287.273880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7288.802581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7288.802694] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7288.808756] Lustre: Skipped 1 previous similar message [ 7288.814317] Lustre: Skipped 1 previous similar message [ 7291.672547] Lustre: server umount lustre-MDT0000 complete [ 7292.816164] LustreError: 335110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778930015 with bad export cookie 18406804351279985169 [ 7292.818457] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7292.819709] LustreError: 335110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7292.921106] Lustre: server umount lustre-MDT0001 complete [ 7303.385639] Lustre: server umount lustre-OST0000 complete [ 7307.675286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7307.827292] LustreError: 338517:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7307.835913] LustreError: 338517:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 7307.848741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7309.160089] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7311.987630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7312.106710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7313.364887] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7315.221967] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7316.423414] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7318.637405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7320.583124] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7322.261722] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7334.121738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 7348.192744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7348.196507] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7348.200659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7351.072062] Lustre: server umount lustre-MDT0000 complete [ 7352.283992] LustreError: 338499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778930074 with bad export cookie 18406804351279986268 [ 7352.286568] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7352.287262] LustreError: 338499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7352.389499] Lustre: server umount lustre-MDT0001 complete [ 7363.867414] Lustre: server umount lustre-OST0000 complete [ 7368.846031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7368.979928] LustreError: 342106:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7369.002639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7369.004995] Lustre: Skipped 1 previous similar message [ 7370.208850] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7372.778499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7374.116684] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7375.864768] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7377.097199] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7379.363353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7381.505081] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7383.316833] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7393.833376] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 7397.856963] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7397.859601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7397.864427] Lustre: Skipped 2 previous similar messages [ 7397.866241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7397.867794] Lustre: Skipped 3 previous similar messages [ 7401.109789] Lustre: server umount lustre-OST0000 complete [ 7403.488710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7403.490915] LustreError: Skipped 1 previous similar message [ 7403.492263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7403.495361] Lustre: Skipped 1 previous similar message [ 7403.496858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7403.498382] Lustre: Skipped 2 previous similar messages [ 7408.472636] Lustre: server umount lustre-MDT0000 complete [ 7409.120995] LustreError: 342107:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7409.127343] LustreError: 342107:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 7409.699306] LustreError: 342086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778930132 with bad export cookie 18406804351279987276 [ 7409.703381] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7409.703939] LustreError: 342086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7409.826833] Lustre: server umount lustre-MDT0001 complete [ 7413.295270] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 7414.358406] Key type lgssc unregistered [ 7414.492410] LNet: 345299:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7414.494763] LNetError: 345299:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7414.505461] LNet: Removed LNI 192.168.203.116@tcp [ 7414.804407] Key type .llcrypt unregistered [ 7414.806042] Key type ._llcrypt unregistered [ 7423.630866] Key type ._llcrypt registered [ 7423.631844] Key type .llcrypt registered [ 7423.668060] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7423.982878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7423.992280] alg: No test for adler32 (adler32-zlib) [ 7424.838240] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 7424.920571] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 7426.496153] Key type lgssc registered [ 7426.870360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7429.308586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7429.314589] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7429.317362] LustreError: 346699:0:(osd_handler.c:8550:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 7429.327058] LustreError: 346699:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 7429.329191] LustreError: 346699:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 7429.331246] LustreError: 346699:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 7429.333500] LustreError: 346699:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7431.737297] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7431.741207] LustreError: 347096:0:(osd_handler.c:8550:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 7431.750904] LustreError: 347096:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 7431.753542] LustreError: 347096:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 7431.755720] LustreError: 347096:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 7431.758080] LustreError: 347096:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7437.782369] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 7438.823971] Key type lgssc unregistered [ 7438.971358] LNet: 348081:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7438.974357] LNetError: 348081:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7438.981397] LNet: Removed LNI 192.168.203.116@tcp [ 7439.266939] Key type .llcrypt unregistered [ 7439.268158] Key type ._llcrypt unregistered [ 7444.949570] Key type ._llcrypt registered [ 7444.951063] Key type .llcrypt registered [ 7444.986873] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 7450.869297] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7451.287244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7451.310747] alg: No test for adler32 (adler32-zlib) [ 7452.151872] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 7452.220940] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 7453.800134] Key type lgssc registered [ 7454.127411] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7464.861334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7464.868708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7465.959717] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7465.973092] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7466.010980] Lustre: lustre-MDT0000: new disk, initializing [ 7466.038865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7466.049612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7467.313251] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7471.314094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7471.362481] Lustre: 351176:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7471.382671] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7471.384325] Lustre: Skipped 1 previous similar message [ 7471.427626] Lustre: lustre-MDT0001: new disk, initializing [ 7471.459394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7471.472571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7471.475834] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7472.734345] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7475.743217] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7476.882271] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7478.987426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7479.084935] Lustre: lustre-OST0000: new disk, initializing [ 7479.086910] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7479.089192] Lustre: 352918:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7479.112866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7480.634585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7480.638260] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7480.675527] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7480.905032] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7483.875416] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7485.217154] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7485.293963] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7486.558011] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7486.627244] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7491.041491] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7491.042894] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7491.043749] LustreError: Skipped 1 previous similar message [ 7491.049098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7491.051267] Lustre: Skipped 1 previous similar message [ 7493.334996] Lustre: server umount lustre-OST0000 complete [ 7496.162555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7496.163277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7496.167826] Lustre: Skipped 2 previous similar messages [ 7500.696061] Lustre: server umount lustre-MDT0000 complete [ 7501.956176] LustreError: 351170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778930224 with bad export cookie 18180872705782288773 [ 7501.958620] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7501.960985] LustreError: 351170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7502.083188] Lustre: server umount lustre-MDT0001 complete [ 7504.871454] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 7505.401971] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 07:17:07 (1778930227) [ 7507.567914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7507.705758] LustreError: 355228:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7507.712276] LustreError: 355228:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 7507.726816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7508.947871] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7511.607463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7511.727900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7512.936626] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7514.573643] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7515.679935] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7517.914327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7519.984858] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7521.846309] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7524.115533] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7524.181531] Lustre: lustre-OST0001: new disk, initializing [ 7524.183511] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7524.186385] Lustre: 357718:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7524.216465] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7524.218694] Lustre: Skipped 1 previous similar message [ 7526.082602] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7529.106223] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7531.001786] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7531.003320] Lustre: Skipped 1 previous similar message [ 7535.716973] Lustre: lustre-OST0001: Not available for connect from 192.168.203.16@tcp (stopping) [ 7535.720694] Lustre: Skipped 2 previous similar messages [ 7536.955627] Lustre: server umount lustre-OST0001 complete [ 7541.219292] LustreError: 356819:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7541.224102] LustreError: 356819:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 7546.340036] LustreError: 356820:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7546.344895] LustreError: 356820:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 7549.920931] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7549.924308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7549.929063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7549.931063] Lustre: Skipped 2 previous similar messages [ 7551.458225] LustreError: 356821:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7551.464596] LustreError: 356821:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 7556.579185] LustreError: 356820:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7556.585123] LustreError: 356820:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 7561.700292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7561.702903] Lustre: Skipped 5 previous similar messages [ 7563.232178] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7563.290395] Lustre: server umount lustre-OST0000 complete [ 7566.817536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7566.820716] Lustre: Skipped 1 previous similar message [ 7570.776845] Lustre: server umount lustre-MDT0000 complete [ 7571.936821] LustreError: 355247:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7571.942893] LustreError: 355247:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 7572.019369] LustreError: 355209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778930294 with bad export cookie 18180872705782289641 [ 7572.020683] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7572.025072] LustreError: 355209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7578.230737] Lustre: server umount lustre-MDT0001 complete [ 7582.062148] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 7583.192417] Key type lgssc unregistered [ 7583.331372] LNet: 359573:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7583.333887] LNetError: 359573:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7583.340316] LNet: Removed LNI 192.168.203.116@tcp [ 7583.650268] Key type .llcrypt unregistered [ 7583.651363] Key type ._llcrypt unregistered [ 7601.186200] Key type ._llcrypt registered [ 7601.187380] Key type .llcrypt registered [ 7601.217311] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7601.623824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7601.635730] alg: No test for adler32 (adler32-zlib) [ 7602.481688] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 7602.560084] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 7604.136181] Key type lgssc registered [ 7604.442107] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7606.758129] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7606.763288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7607.847451] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7607.855894] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7607.912804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7609.103232] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7611.756053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7611.790041] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7611.799192] Lustre: 362330:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7611.883066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7613.102044] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7614.820501] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7615.886356] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7616.994370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7616.995277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7616.999546] Lustre: Skipped 1 previous similar message [ 7617.003423] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7622.112831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7622.115644] Lustre: Skipped 2 previous similar messages [ 7622.673855] Lustre: server umount lustre-MDT0000 complete [ 7623.778427] LustreError: 362322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778930346 with bad export cookie 2403288739766732568 [ 7623.781051] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7623.782203] LustreError: 362322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7623.886143] Lustre: server umount lustre-MDT0001 complete [ 7626.626050] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 07:19:08 (1778930348) [ 7628.635742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7628.789900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7629.933974] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7632.487969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7632.610075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7633.945481] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7635.747145] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7636.946476] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7639.192161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7639.261110] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7639.330161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7641.227109] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7643.035363] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7651.925769] Lustre: Permanently deactivating lustre-OST0000 [ 7651.931527] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7653.234511] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7653.841480] LustreError: 367109:0:(mgs_llog.c:2031:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 7653.847292] LustreError: 367109:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 7653.851124] LustreError: 367109:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7654.518654] LustreError: 367157:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 7654.522230] LustreError: 367157:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7661.269412] Lustre: server umount lustre-OST0000 complete [ 7662.165511] LustreError: 367406:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 7662.168886] LustreError: 367406:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7663.584754] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7663.588036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7663.592565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7665.122368] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7670.240971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7670.243241] Lustre: Skipped 3 previous similar messages [ 7673.333102] Lustre: server umount lustre-MDT0000 complete [ 7674.512036] LustreError: 364589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778930397 with bad export cookie 2403288739766733065 [ 7674.514603] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7674.516459] LustreError: 364589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7685.142711] Lustre: server umount lustre-MDT0001 complete [ 7688.071320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7689.490414] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7690.378628] LustreError: 368546:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 7692.483763] Lustre: 368884:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.116@tcp [ 7693.200949] Lustre: server umount MGS complete [ 7697.025248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7697.198182] LustreError: 369607:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7697.203610] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7697.203741] LustreError: 369607:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 7697.206404] Lustre: Skipped 1 previous similar message [ 7697.229955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7698.525148] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7701.364716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7701.469188] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7702.881805] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7704.745215] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7705.967305] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7706.562348] Lustre: Permanently reactivating lustre-OST0000 [ 7706.567025] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7706.569169] Lustre: Skipped 2 previous similar messages [ 7727.072151] LustreError: 370960:0:(osp_dev.c:695:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 7729.821388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7729.951063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7729.954437] Lustre: Skipped 1 previous similar message [ 7731.847070] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7733.562268] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7747.555606] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7747.556650] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7747.557698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7747.567497] Lustre: Skipped 1 previous similar message [ 7751.500594] Lustre: server umount lustre-OST0000 complete [ 7757.793746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7757.793850] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7757.794334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7757.794339] Lustre: Skipped 1 previous similar message [ 7757.798274] Lustre: Skipped 1 previous similar message [ 7757.802042] LustreError: Skipped 1 previous similar message [ 7758.936152] Lustre: server umount lustre-MDT0000 complete [ 7760.226209] LustreError: 369587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778930482 with bad export cookie 2403288739766734087 [ 7760.228852] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7760.229577] LustreError: 369587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7760.365180] Lustre: server umount lustre-MDT0001 complete [ 7764.087326] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 7765.216722] Key type lgssc unregistered [ 7765.359395] LNet: 373731:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7765.362317] LNetError: 373731:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7765.371366] LNet: Removed LNI 192.168.203.116@tcp [ 7765.684261] Key type .llcrypt unregistered [ 7765.685565] Key type ._llcrypt unregistered [ 7771.709378] Key type ._llcrypt registered [ 7771.710484] Key type .llcrypt registered [ 7771.744355] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 7777.920424] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7778.229518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7778.239427] alg: No test for adler32 (adler32-zlib) [ 7779.097054] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 7779.189894] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 7780.768166] Key type lgssc registered [ 7781.128968] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7792.439625] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7792.448437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7793.542650] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7793.553506] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7793.596978] Lustre: lustre-MDT0000: new disk, initializing [ 7793.622652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7793.629885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7794.969424] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7799.060616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7799.100731] Lustre: 376826:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7799.126730] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7799.129791] Lustre: Skipped 1 previous similar message [ 7799.173676] Lustre: lustre-MDT0001: new disk, initializing [ 7799.210403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7799.221325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7799.224467] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7800.479631] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7803.596605] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7804.838161] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7807.127220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7807.223695] Lustre: lustre-OST0000: new disk, initializing [ 7807.225418] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7807.228145] Lustre: 378568:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7807.256914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7809.207628] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7812.368839] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7813.732278] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7816.192694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7816.196993] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7816.231550] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7816.874707] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 7818.239265] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7818.312169] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7821.283923] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7821.283994] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7821.286779] LustreError: Skipped 1 previous similar message [ 7821.292872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7821.296258] Lustre: Skipped 1 previous similar message [ 7825.046689] Lustre: server umount lustre-OST0000 complete [ 7826.401650] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7826.402216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7826.405804] Lustre: Skipped 2 previous similar messages [ 7831.521481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7831.523296] Lustre: Skipped 2 previous similar messages [ 7832.474539] Lustre: server umount lustre-MDT0000 complete [ 7833.703269] LustreError: 376817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778930556 with bad export cookie 11781717155801850571 [ 7833.706176] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7833.707329] LustreError: 376817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7833.825117] Lustre: server umount lustre-MDT0001 complete [ 7836.494832] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 7837.008884] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 07:22:39 (1778930559) [ 7839.102305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7839.275938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7840.510894] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7843.319714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7843.442223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7844.759599] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7846.596174] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7847.785331] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7850.071625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7852.143112] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7853.922684] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7854.553468] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 7865.503827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 7868.899039] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7868.899561] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7868.900736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7868.908811] Lustre: Skipped 1 previous similar message [ 7873.109324] Lustre: server umount lustre-OST0000 complete [ 7874.528722] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7874.531201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7874.535663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7874.537459] Lustre: Skipped 1 previous similar message [ 7879.137663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7880.470809] Lustre: server umount lustre-MDT0000 complete [ 7881.801250] LustreError: 380880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778930604 with bad export cookie 11781717155801851439 [ 7881.804572] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7881.805631] LustreError: 380880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7881.928208] Lustre: server umount lustre-MDT0001 complete [ 7885.642777] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 7886.695208] Key type lgssc unregistered [ 7886.829437] LNet: 384290:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7886.832234] LNetError: 384290:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7886.841288] LNet: Removed LNI 192.168.203.116@tcp [ 7887.144175] Key type .llcrypt unregistered [ 7887.145708] Key type ._llcrypt unregistered [ 7894.675775] Key type ._llcrypt registered [ 7894.676883] Key type .llcrypt registered [ 7894.713399] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 7894.806251] Key type .llcrypt unregistered [ 7894.807636] Key type ._llcrypt unregistered [ 7901.571015] Key type ._llcrypt registered [ 7901.572034] Key type .llcrypt registered [ 7901.609875] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7901.967474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7901.998521] alg: No test for adler32 (adler32-zlib) [ 7902.849361] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 7902.941285] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 7904.528097] Key type lgssc registered [ 7904.879061] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7907.399945] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7907.406636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7908.562742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7909.844683] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7912.590326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7912.701530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7914.603605] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7916.392900] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7918.049706] LustreError: 386444:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7918.057766] LustreError: 386444:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 7918.617835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7918.735909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7919.785813] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7920.068083] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7924.193430] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7924.196994] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7924.203260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7925.219470] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7925.221560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7925.222662] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7928.979073] Lustre: server umount lustre-OST0000 complete [ 7930.337299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7930.337960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7930.341590] Lustre: Skipped 1 previous similar message [ 7930.347390] Lustre: Skipped 2 previous similar messages [ 7935.456969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7935.459789] Lustre: Skipped 1 previous similar message [ 7936.282638] Lustre: server umount lustre-MDT0000 complete [ 7937.493150] LustreError: 388054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778930660 with bad export cookie 8937586658035343439 [ 7937.497072] LustreError: 388054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7937.497238] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7937.614832] Lustre: server umount lustre-MDT0001 complete [ 7941.261460] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 7942.221160] Key type lgssc unregistered [ 7942.345334] LNet: 389425:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7942.348292] LNetError: 389425:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7942.358305] LNet: Removed LNI 192.168.203.116@tcp [ 7942.644564] Key type .llcrypt unregistered [ 7942.645739] Key type ._llcrypt unregistered [ 7951.835849] Key type ._llcrypt registered [ 7951.836933] Key type .llcrypt registered [ 7951.876627] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7952.360872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7952.373578] alg: No test for adler32 (adler32-zlib) [ 7953.216532] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 7953.298239] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 7954.880134] Key type lgssc registered [ 7955.210147] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7957.721365] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7957.728231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7973.152346] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7978.272231] LustreError: 390775:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.116@tcp: failed processing log, type 4: rc = -110 [ 8004.896195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8006.731189] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8008.556913] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8010.804814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8010.996875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8011.027150] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8012.296166] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8015.045226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8015.166555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8016.408217] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8018.162585] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8019.393460] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8026.592867] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8026.596592] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8026.602811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8030.692993] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8030.693557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8030.695468] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8031.893578] Lustre: server umount lustre-OST0000 complete [ 8035.809815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8035.810113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8035.817503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8035.819375] Lustre: Skipped 2 previous similar messages [ 8039.318702] Lustre: server umount lustre-MDT0000 complete [ 8040.526338] LustreError: 391726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778930763 with bad export cookie 17373062576954935109 [ 8040.530537] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8040.531416] LustreError: 391726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8040.661370] Lustre: server umount lustre-MDT0001 complete [ 8044.115682] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 8045.087683] Key type lgssc unregistered [ 8045.220349] LNet: 393988:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8045.224318] LNetError: 393988:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8045.232309] LNet: Removed LNI 192.168.203.116@tcp [ 8045.493387] Key type .llcrypt unregistered [ 8045.495038] Key type ._llcrypt unregistered [ 8054.510673] Key type ._llcrypt registered [ 8054.511757] Key type .llcrypt registered [ 8054.550462] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 8054.873732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8054.913056] alg: No test for adler32 (adler32-zlib) [ 8055.764357] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 8055.849183] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 8057.432144] Key type lgssc registered [ 8057.794930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8060.247485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8060.255471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8061.419688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8062.669654] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8065.373869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8065.497618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8066.734641] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8068.395106] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8069.492693] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8071.603906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8071.651350] LustreError: 396955:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8071.656462] LustreError: 396955:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 8071.714354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8073.458115] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8075.122231] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8075.814594] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8085.986628] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8085.987812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8085.988681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8085.990472] Lustre: Skipped 2 previous similar messages [ 8086.626962] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.16@tcp (stopping) [ 8086.629106] Lustre: Skipped 2 previous similar messages [ 8088.282818] Lustre: server umount lustre-MDT0000 complete [ 8089.018805] LustreError: 395343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.16@tcp arrived at 1778930811 with bad export cookie 9998760833565194407 [ 8089.022782] LustreError: 395343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8089.645259] LustreError: 395344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778930812 with bad export cookie 9998760833565193763 [ 8089.649385] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8091.105163] LustreError: 395363:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8091.105674] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8091.105971] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8091.106544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8091.109917] LustreError: 395363:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 8094.112902] LustreError: 396086:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8094.113227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8094.118974] LustreError: 396086:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 8095.765656] Lustre: server umount lustre-OST0000 complete [ 8098.004621] Lustre: server umount lustre-MDT0001 complete [ 8101.357610] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 8102.326671] Key type lgssc unregistered [ 8102.454349] LNet: 398687:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8102.457308] LNetError: 398687:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8102.466327] LNet: Removed LNI 192.168.203.116@tcp [ 8102.752352] Key type .llcrypt unregistered [ 8102.753384] Key type ._llcrypt unregistered [ 8111.609722] Key type ._llcrypt registered [ 8111.610823] Key type .llcrypt registered [ 8111.657814] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 8112.161636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8112.185759] alg: No test for adler32 (adler32-zlib) [ 8113.042089] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 8113.120054] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 8114.696137] Key type lgssc registered [ 8115.070531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8117.571651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8117.578347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8118.738421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8119.972459] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8122.725588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8122.838099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8123.997763] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8125.729678] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8126.854416] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8129.075181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8129.195377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8131.039735] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8132.777338] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8143.526128] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8153.568971] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8153.571382] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8153.577476] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8154.081231] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8154.083063] Lustre: Skipped 1 previous similar message [ 8154.083525] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8155.604049] Lustre: server umount lustre-MDT0001 complete [ 8159.201353] LustreError: 400772:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8159.202287] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8159.202653] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8159.206831] LustreError: 400772:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 8159.211783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8159.217806] Lustre: Skipped 1 previous similar message [ 8163.351116] Lustre: server umount lustre-OST0000 complete [ 8164.323649] LustreError: 400062:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8164.724642] Lustre: server umount lustre-MDT0000 complete [ 8169.016429] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 8170.039876] Key type lgssc unregistered [ 8170.162491] LNet: 403413:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8170.165291] LNetError: 403413:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8170.174513] LNet: Removed LNI 192.168.203.116@tcp [ 8170.474932] Key type .llcrypt unregistered [ 8170.476037] Key type ._llcrypt unregistered [ 8177.992472] Key type ._llcrypt registered [ 8177.993613] Key type .llcrypt registered [ 8178.026653] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 8183.824831] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 8184.289362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8184.295346] alg: No test for adler32 (adler32-zlib) [ 8185.144062] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 8185.224534] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 8186.800206] Key type lgssc registered [ 8187.132202] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8198.664294] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 8203.453784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8203.458397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8204.540504] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8204.551348] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8204.590346] Lustre: lustre-MDT0000: new disk, initializing [ 8204.614705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8204.622768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8205.884177] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8211.296749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8211.386736] Lustre: lustre-OST0000: new disk, initializing [ 8211.388315] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8211.389983] Lustre: Skipped 1 previous similar message [ 8211.392959] Lustre: 408349:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8211.414088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8211.816680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8211.819487] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8211.827799] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8213.037644] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8215.980989] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8221.324298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8221.370208] Lustre: 407348:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8221.395970] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8221.434717] Lustre: lustre-MDT0001: new disk, initializing [ 8221.476179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8221.488899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 8221.493579] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 8222.798676] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8226.263292] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 8226.334068] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8227.633408] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 8227.706348] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 8231.315191] Lustre: server umount lustre-MDT0001 complete [ 8232.417724] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8232.418845] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8232.425285] Lustre: Skipped 1 previous similar message [ 8237.539189] LustreError: 407355:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8237.546938] LustreError: 407355:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 8242.658974] LustreError: 407354:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8242.664625] LustreError: 407354:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 8247.777845] LustreError: 409561:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8247.778340] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8247.779048] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8247.779444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8249.684290] Lustre: server umount lustre-OST0000 complete [ 8257.143957] Lustre: server umount lustre-MDT0000 complete [ 8261.526429] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 8262.459372] Key type lgssc unregistered [ 8262.585384] LNet: 411910:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8262.589224] LNetError: 411910:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8262.599339] LNet: Removed LNI 192.168.203.116@tcp [ 8262.867366] Key type .llcrypt unregistered [ 8262.868537] Key type ._llcrypt unregistered [ 8275.409168] Key type ._llcrypt registered [ 8275.410280] Key type .llcrypt registered [ 8275.443707] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 8275.531166] Key type .llcrypt unregistered [ 8275.532259] Key type ._llcrypt unregistered [ 8282.064796] Key type ._llcrypt registered [ 8282.066017] Key type .llcrypt registered [ 8282.100975] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 8282.407168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8282.481035] alg: No test for adler32 (adler32-zlib) [ 8283.337170] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 8283.416091] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 8285.000100] Key type lgssc registered [ 8285.331714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8294.011980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8294.020932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8295.131659] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8295.144145] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8295.180307] Lustre: lustre-MDT0000: new disk, initializing [ 8295.207781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8295.214083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8296.606846] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8300.722705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8300.758188] Lustre: 415222:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8300.779153] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8300.781082] Lustre: Skipped 1 previous similar message [ 8300.818834] Lustre: lustre-MDT0001: new disk, initializing [ 8300.821300] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8300.848574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8300.859438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8300.862933] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8302.014147] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8304.893461] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8306.024452] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8308.074363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8308.167988] Lustre: lustre-OST0000: new disk, initializing [ 8308.170587] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8308.172889] Lustre: 416963:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8308.199082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8308.299671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8308.302476] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8308.329489] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8309.924575] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8312.915904] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8316.384807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8316.387233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8316.391162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8318.434570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8318.437734] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8321.310652] Lustre: server umount lustre-MDT0000 complete [ 8322.515025] LustreError: 416962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778931045 with bad export cookie 5583519792581426278 [ 8322.518547] LustreError: 416962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8322.518694] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8322.630056] Lustre: server umount lustre-MDT0001 complete [ 8334.100505] Lustre: server umount lustre-OST0000 complete [ 8337.365167] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 07:30:59 (1778931059) [ 8339.861880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8340.046923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8341.345251] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8343.941296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8344.057127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8345.277609] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8347.084868] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8348.283451] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8363.619188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8363.682832] Lustre: Found index 0 for lustre-OST0000, updating log [ 8363.734289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8364.774099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 8365.521635] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8367.188336] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8369.250608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8369.251038] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8369.252425] Lustre: Skipped 2 previous similar messages [ 8369.256262] Lustre: Skipped 1 previous similar message [ 8374.351068] Lustre: server umount lustre-OST0000 complete [ 8390.113054] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8390.115378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8390.119350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8401.376204] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8401.472983] Lustre: server umount lustre-MDT0000 complete [ 8402.402074] LustreError: 419589:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8402.412024] LustreError: 419589:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 8402.812552] LustreError: 418902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778931125 with bad export cookie 5583519792581427636 [ 8402.815251] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8402.816500] LustreError: 418902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8402.947245] Lustre: server umount lustre-MDT0001 complete [ 8406.021993] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 07:32:07 (1778931127) [ 8432.928401] LNet: 1 local NIs in recovery (showing 1): 192.168.203.116@tcp [ 8987.668557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8987.833755] LustreError: 424942:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8987.860819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8989.130308] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8992.154826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8993.708994] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8995.965187] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8997.374925] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9012.581182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9012.686679] Lustre: Found index 0 for lustre-OST0000, updating log [ 9014.792855] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9015.850501] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 9017.162840] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9018.915320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9018.915405] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9018.917990] Lustre: Skipped 6 previous similar messages [ 9018.923512] Lustre: Skipped 1 previous similar message [ 9024.015335] Lustre: server umount lustre-OST0000 complete [ 9026.339846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9026.342465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9026.346776] Lustre: Skipped 1 previous similar message [ 9026.353725] Lustre: Skipped 1 previous similar message [ 9028.385214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9028.387180] Lustre: Skipped 3 previous similar messages [ 9031.452767] Lustre: server umount lustre-MDT0000 complete [ 9031.458474] LustreError: 424943:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9031.463956] LustreError: 424943:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 9033.096538] LustreError: 424924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778931755 with bad export cookie 5583519792581428700 [ 9033.101572] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9033.102284] LustreError: 424924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9033.222317] Lustre: server umount lustre-MDT0001 complete [ 9046.305737] LNet: 1 local NIs in recovery (showing 1): 192.168.203.116@tcp [ 9051.336922] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 9055.011369] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 9070.960923] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 07:43:12 (1778931792) [ 9077.723600] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 9078.995078] Key type lgssc unregistered [ 9079.159559] LNet: 431205:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9079.162913] LNetError: 431205:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9143.718538] LNet: Removed LNI 192.168.203.116@tcp [ 9144.072135] Key type .llcrypt unregistered [ 9144.074385] Key type ._llcrypt unregistered [ 9152.507601] Key type ._llcrypt registered [ 9152.509300] Key type .llcrypt registered [ 9152.567271] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 9152.955639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9152.970938] alg: No test for adler32 (adler32-zlib) [ 9153.850544] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 9153.950234] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 9155.544180] Key type lgssc registered [ 9156.065637] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9158.085351] LNet: Added LNI 192.168.203.116@tcp100 [8/256/0/180] [ 9158.455134] LNet: Added LNI 192.168.203.116@tcp101 [8/256/0/180] [ 9158.839059] LNet: Added LNI 192.168.203.116@tcp102 [8/256/0/180] [ 9159.255793] LNet: Added LNI 192.168.203.116@tcp103 [8/256/0/180] [ 9159.669503] LNet: Added LNI 192.168.203.116@tcp104 [8/256/0/180] [ 9160.084684] LNet: Added LNI 192.168.203.116@tcp105 [8/256/0/180] [ 9160.469649] LNet: Added LNI 192.168.203.116@tcp106 [8/256/0/180] [ 9160.900464] LNet: Added LNI 192.168.203.116@tcp107 [8/256/0/180] [ 9161.288533] LNet: Added LNI 192.168.203.116@tcp108 [8/256/0/180] [ 9161.710055] LNet: Added LNI 192.168.203.116@tcp109 [8/256/0/180] [ 9162.102455] LNet: Added LNI 192.168.203.116@tcp110 [8/256/0/180] [ 9162.556102] LNet: Added LNI 192.168.203.116@tcp111 [8/256/0/180] [ 9162.984871] LNet: Added LNI 192.168.203.116@tcp112 [8/256/0/180] [ 9163.459794] LNet: Added LNI 192.168.203.116@tcp113 [8/256/0/180] [ 9163.894479] LNet: Added LNI 192.168.203.116@tcp114 [8/256/0/180] [ 9164.354507] LNet: Added LNI 192.168.203.116@tcp115 [8/256/0/180] [ 9164.796662] LNet: Added LNI 192.168.203.116@tcp116 [8/256/0/180] [ 9165.260561] LNet: Added LNI 192.168.203.116@tcp117 [8/256/0/180] [ 9165.754867] LNet: Added LNI 192.168.203.116@tcp118 [8/256/0/180] [ 9166.209322] LNet: Added LNI 192.168.203.116@tcp119 [8/256/0/180] [ 9166.674618] LNet: Added LNI 192.168.203.116@tcp120 [8/256/0/180] [ 9167.133836] LNet: Added LNI 192.168.203.116@tcp121 [8/256/0/180] [ 9167.597781] LNet: Added LNI 192.168.203.116@tcp122 [8/256/0/180] [ 9168.054183] LNet: Added LNI 192.168.203.116@tcp123 [8/256/0/180] [ 9168.522582] LNet: Added LNI 192.168.203.116@tcp124 [8/256/0/180] [ 9168.972308] LNet: Added LNI 192.168.203.116@tcp125 [8/256/0/180] [ 9169.372711] LNet: Added LNI 192.168.203.116@tcp126 [8/256/0/180] [ 9169.795835] LNet: Added LNI 192.168.203.116@tcp127 [8/256/0/180] [ 9170.238873] LNet: Added LNI 192.168.203.116@tcp128 [8/256/0/180] [ 9170.677806] LNet: Added LNI 192.168.203.116@tcp129 [8/256/0/180] [ 9171.038672] LNet: Added LNI 192.168.203.116@tcp130 [8/256/0/180] [ 9171.402837] LNet: Added LNI 192.168.203.116@tcp131 [8/256/0/180] [ 9171.826537] LNet: Added LNI 192.168.203.116@tcp132 [8/256/0/180] [ 9172.192319] LNet: Added LNI 192.168.203.116@tcp133 [8/256/0/180] [ 9172.631802] LNet: Added LNI 192.168.203.116@tcp134 [8/256/0/180] [ 9173.080891] LNet: Added LNI 192.168.203.116@tcp135 [8/256/0/180] [ 9176.132544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9176.145152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9177.209214] Lustre: 434186:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.116@tcp: can't add NID 192.168.203.116@tcp130: rc = -75 [ 9177.291352] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9177.305882] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9177.350990] Lustre: lustre-MDT0000: new disk, initializing [ 9177.396876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9177.408940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9179.192885] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9184.548306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9184.609335] Lustre: 434203:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9184.642848] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9184.645586] Lustre: Skipped 1 previous similar message [ 9184.686374] Lustre: lustre-MDT0001: new disk, initializing [ 9184.698884] Lustre: 434986:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.116@tcp: can't add NID 192.168.203.116@tcp130: rc = -75 [ 9184.701338] Lustre: 434986:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 9184.762155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9184.784415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9184.791103] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9186.302384] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9189.377387] Lustre: 435443:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.116@tcp: can't add NID 192.168.203.116@tcp130: rc = -75 [ 9189.384393] Lustre: 435443:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 9190.027667] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9191.674746] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9194.815708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9194.956089] Lustre: lustre-OST0000: new disk, initializing [ 9194.961806] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9194.967215] Lustre: 435949:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9195.018393] Lustre: 435954:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.116@tcp: can't add NID 192.168.203.116@tcp130: rc = -75 [ 9195.023408] Lustre: 435954:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 9195.047486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9197.559761] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9197.787867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9197.793767] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9197.877802] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9201.151229] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9208.293519] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9208.294861] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9208.295604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9208.295714] LustreError: Skipped 1 previous similar message [ 9208.298894] Lustre: Skipped 1 previous similar message [ 9209.936689] Lustre: server umount lustre-OST0000 complete [ 9213.411920] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9213.413765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9213.420181] Lustre: Skipped 1 previous similar message [ 9213.425182] Lustre: Skipped 1 previous similar message [ 9217.282836] Lustre: server umount lustre-MDT0000 complete [ 9219.125062] LustreError: 434196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778931941 with bad export cookie 4611584300262157885 [ 9219.129625] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9219.132784] LustreError: 434196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9219.291194] Lustre: server umount lustre-MDT0001 complete [ 9223.803758] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 9225.346887] Key type lgssc unregistered [ 9225.572938] LNet: 437727:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9225.579296] LNetError: 437727:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9225.584350] LNet: Removed LNI 192.168.203.116@tcp [ 9226.592206] LNet: Removed LNI 192.168.203.116@tcp100 [ 9226.593918] LNetError: 437727:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9226.596647] LNetError: 437727:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 9226.600259] LNet: Removed LNI 192.168.203.116@tcp101 [ 9227.616096] LNet: Removed LNI 192.168.203.116@tcp102 [ 9227.618470] LNetError: 437727:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9227.622554] LNetError: 437727:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 9227.628029] LNet: Removed LNI 192.168.203.116@tcp103 [ 9228.640176] LNet: Removed LNI 192.168.203.116@tcp104 [ 9229.664181] LNet: Removed LNI 192.168.203.116@tcp105 [ 9229.668314] LNetError: 437727:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9229.675185] LNetError: 437727:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 9229.681837] LNet: Removed LNI 192.168.203.116@tcp106 [ 9230.688214] LNet: Removed LNI 192.168.203.116@tcp107 [ 9231.712221] LNet: Removed LNI 192.168.203.116@tcp108 [ 9232.736115] LNet: Removed LNI 192.168.203.116@tcp109 [ 9233.760194] LNet: Removed LNI 192.168.203.116@tcp110 [ 9233.763437] LNetError: 437727:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9233.770417] LNetError: 437727:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 9233.776465] LNet: Removed LNI 192.168.203.116@tcp111 [ 9234.784184] LNet: Removed LNI 192.168.203.116@tcp112 [ 9235.808167] LNet: Removed LNI 192.168.203.116@tcp113 [ 9236.832258] LNet: Removed LNI 192.168.203.116@tcp114 [ 9236.835222] LNet: Removed LNI 192.168.203.116@tcp115 [ 9236.838061] LNet: Removed LNI 192.168.203.116@tcp116 [ 9236.840678] LNet: Removed LNI 192.168.203.116@tcp117 [ 9236.843460] LNet: Removed LNI 192.168.203.116@tcp118 [ 9236.846243] LNet: Removed LNI 192.168.203.116@tcp119 [ 9236.848856] LNet: Removed LNI 192.168.203.116@tcp120 [ 9236.851522] LNet: Removed LNI 192.168.203.116@tcp121 [ 9236.854286] LNet: Removed LNI 192.168.203.116@tcp122 [ 9236.857092] LNet: Removed LNI 192.168.203.116@tcp123 [ 9236.859738] LNet: Removed LNI 192.168.203.116@tcp124 [ 9236.862437] LNet: Removed LNI 192.168.203.116@tcp125 [ 9236.865198] LNet: Removed LNI 192.168.203.116@tcp126 [ 9236.867942] LNet: Removed LNI 192.168.203.116@tcp127 [ 9236.870528] LNet: Removed LNI 192.168.203.116@tcp128 [ 9236.873233] LNet: Removed LNI 192.168.203.116@tcp129 [ 9236.875950] LNet: Removed LNI 192.168.203.116@tcp130 [ 9236.878575] LNet: Removed LNI 192.168.203.116@tcp131 [ 9236.881311] LNet: Removed LNI 192.168.203.116@tcp132 [ 9236.884028] LNet: Removed LNI 192.168.203.116@tcp133 [ 9236.886599] LNet: Removed LNI 192.168.203.116@tcp134 [ 9236.895915] LNet: Removed LNI 192.168.203.116@tcp135 [ 9237.327864] Key type .llcrypt unregistered [ 9237.330345] Key type ._llcrypt unregistered [ 9246.926613] Key type ._llcrypt registered [ 9246.927946] Key type .llcrypt registered [ 9246.982291] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 9253.825485] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 9254.220241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9254.228086] alg: No test for adler32 (adler32-zlib) [ 9255.123091] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 9255.232876] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 9256.840275] Key type lgssc registered [ 9257.482380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9273.036950] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 07:46:34 (1778931994) [ 9279.636251] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 9281.091392] Key type lgssc unregistered [ 9281.270744] LNet: 441735:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9281.275723] LNetError: 441735:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9281.287713] LNet: Removed LNI 192.168.203.116@tcp [ 9281.752655] Key type .llcrypt unregistered [ 9281.755074] Key type ._llcrypt unregistered [ 9289.796326] Key type ._llcrypt registered [ 9289.798269] Key type .llcrypt registered [ 9289.863330] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 9290.227419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9290.234248] alg: No test for adler32 (adler32-zlib) [ 9291.114162] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 9291.248534] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 9292.856144] Key type lgssc registered [ 9293.362181] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9297.258216] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9297.269994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9298.375852] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9298.387371] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9298.427202] Lustre: lustre-MDT0000: new disk, initializing [ 9298.451775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9298.458220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9299.881859] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9305.061082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9305.107561] Lustre: 442908:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9305.137843] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9305.140950] Lustre: Skipped 1 previous similar message [ 9305.189673] Lustre: lustre-MDT0001: new disk, initializing [ 9305.227269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9305.242466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9305.250248] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9306.522574] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9310.094951] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9311.837530] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9314.723922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9314.893817] Lustre: lustre-OST0000: new disk, initializing [ 9314.898143] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9314.905230] Lustre: 444652:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9314.943725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9316.853503] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9317.261168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9317.267751] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9317.320823] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9320.678665] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9322.089268] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing lnet_if_list [ 9323.935821] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 9323.947640] LNet: Added LNI 192.168.203.116@tcp10 [8/256/0/180] [ 9326.081235] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 9326.100350] LNet: Added LNI 192.168.203.116@tcp11 [8/256/0/180] [ 9327.743986] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 9327.761872] LNet: Added LNI 192.168.203.116@tcp12 [8/256/0/180] [ 9329.452248] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 9329.469666] LNet: Added LNI 192.168.203.116@tcp13 [8/256/0/180] [ 9330.946793] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 9330.963455] LNet: Added LNI 192.168.203.116@tcp14 [8/256/0/180] [ 9332.938883] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 9332.958496] LNet: Added LNI 192.168.203.116@tcp15 [8/256/0/180] [ 9335.036639] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 9335.056495] LNet: Added LNI 192.168.203.116@tcp16 [8/256/0/180] [ 9337.290614] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 9337.311759] LNet: Added LNI 192.168.203.116@tcp17 [8/256/0/180] [ 9339.375929] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 9339.389714] LNet: Added LNI 192.168.203.116@tcp18 [8/256/0/180] [ 9341.132545] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 9341.148597] LNet: Added LNI 192.168.203.116@tcp19 [8/256/0/180] [ 9343.397472] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 9343.418073] LNet: Added LNI 192.168.203.116@tcp20 [8/256/0/180] [ 9345.115597] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 9345.124581] LNet: Added LNI 192.168.203.116@tcp21 [8/256/0/180] [ 9346.457046] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 9346.465690] LNet: Added LNI 192.168.203.116@tcp22 [8/256/0/180] [ 9347.939964] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 9347.953022] LNet: Added LNI 192.168.203.116@tcp23 [8/256/0/180] [ 9349.900284] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 9349.916122] LNet: Added LNI 192.168.203.116@tcp24 [8/256/0/180] [ 9351.923540] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 9351.935531] LNet: Added LNI 192.168.203.116@tcp25 [8/256/0/180] [ 9354.035078] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 9354.046231] LNet: Added LNI 192.168.203.116@tcp26 [8/256/0/180] [ 9355.978298] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 9355.992571] LNet: Added LNI 192.168.203.116@tcp27 [8/256/0/180] [ 9358.231330] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 9358.244787] LNet: Added LNI 192.168.203.116@tcp28 [8/256/0/180] [ 9360.396298] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 9360.417129] LNet: Added LNI 192.168.203.116@tcp29 [8/256/0/180] [ 9362.558956] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 9362.569699] LNet: Added LNI 192.168.203.116@tcp30 [8/256/0/180] [ 9364.556826] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 9364.569611] LNet: Added LNI 192.168.203.116@tcp31 [8/256/0/180] [ 9366.519802] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 9366.534198] LNet: Added LNI 192.168.203.116@tcp32 [8/256/0/180] [ 9368.928432] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 9368.943749] LNet: Added LNI 192.168.203.116@tcp33 [8/256/0/180] [ 9371.146868] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 9371.162704] LNet: Added LNI 192.168.203.116@tcp34 [8/256/0/180] [ 9373.621057] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 9373.633862] LNet: Added LNI 192.168.203.116@tcp35 [8/256/0/180] [ 9375.740288] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 9375.750538] LNet: Added LNI 192.168.203.116@tcp36 [8/256/0/180] [ 9378.332103] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 9378.340752] LNet: Added LNI 192.168.203.116@tcp37 [8/256/0/180] [ 9380.436961] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 9380.447840] LNet: Added LNI 192.168.203.116@tcp38 [8/256/0/180] [ 9382.621451] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 9382.633308] LNet: Added LNI 192.168.203.116@tcp39 [8/256/0/180] [ 9384.973656] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 9384.990062] LNet: Added LNI 192.168.203.116@tcp40 [8/256/0/180] [ 9387.444627] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 9387.454274] LNet: Added LNI 192.168.203.116@tcp41 [8/256/0/180] [ 9389.634660] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 9389.650596] LNet: Added LNI 192.168.203.116@tcp42 [8/256/0/180] [ 9392.388216] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 9392.405405] LNet: Added LNI 192.168.203.116@tcp43 [8/256/0/180] [ 9394.993693] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 9395.013839] LNet: Added LNI 192.168.203.116@tcp44 [8/256/0/180] [ 9397.810243] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 9397.824493] LNet: Added LNI 192.168.203.116@tcp45 [8/256/0/180] [ 9400.521917] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 9400.532658] LNet: Added LNI 192.168.203.116@tcp46 [8/256/0/180] [ 9403.323669] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 9403.341764] LNet: Added LNI 192.168.203.116@tcp47 [8/256/0/180] [ 9406.083278] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 9406.093460] LNet: Added LNI 192.168.203.116@tcp48 [8/256/0/180] [ 9408.622789] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 9408.637609] LNet: Added LNI 192.168.203.116@tcp49 [8/256/0/180] [ 9411.750187] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 9411.761968] LNet: Added LNI 192.168.203.116@tcp50 [8/256/0/180] [ 9423.330082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9423.335912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9423.345939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9424.869802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9424.870125] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9424.874021] Lustre: Skipped 1 previous similar message [ 9424.883450] Lustre: Skipped 1 previous similar message [ 9427.987741] Lustre: server umount lustre-MDT0000 complete [ 9429.821802] LustreError: 442899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778932152 with bad export cookie 3907740956635604060 [ 9429.826617] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9429.829146] LustreError: 442899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9429.973587] Lustre: server umount lustre-MDT0001 complete [ 9438.149794] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 9441.280758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9445.344134] Lustre: 442180:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778932152/real 1778932152] req@00000000711b6b47 x1865345414549632/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778932168 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9445.355072] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9450.464250] Lustre: 442182:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778932157/real 1778932157] req@00000000830853c2 x1865345414549760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778932173 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9454.561758] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x363b15b22a82ac82 [ 9454.567945] Lustre: MGC192.168.203.116@tcp: Connection restored to 0@lo (at 0@lo) [ 9454.748191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9455.584171] Lustre: 442183:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778932162/real 1778932162] req@000000006f324c43 x1865345414549888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778932178 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9456.300692] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9459.633321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9459.767873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9461.198396] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9461.792117] Lustre: 442181:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778932168/real 1778932168] req@00000000c825c816 x1865345414550144/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778932184 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9463.055255] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9464.488168] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9464.805071] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9465.829528] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9475.041619] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9475.044740] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9475.049215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9475.050822] Lustre: Skipped 1 previous similar message [ 9478.822287] Lustre: server umount lustre-OST0000 complete [ 9481.187279] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9481.188380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9481.194570] Lustre: Skipped 2 previous similar messages [ 9481.204499] Lustre: Skipped 4 previous similar messages [ 9486.305946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9486.310167] Lustre: Skipped 1 previous similar message [ 9486.648140] Lustre: server umount lustre-MDT0000 complete [ 9488.119117] LustreError: 443687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778932210 with bad export cookie 3907740956635606146 [ 9488.121727] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9488.122173] LustreError: 443687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9488.230032] Lustre: server umount lustre-MDT0001 complete [ 9492.788933] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 9494.192801] Key type lgssc unregistered [ 9494.416935] LNet: 470528:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9494.422641] LNetError: 470528:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9494.427482] LNet: Removed LNI 192.168.203.116@tcp [ 9494.429711] LNet: Removed LNI 192.168.203.116@tcp10 [ 9494.432375] LNet: Removed LNI 192.168.203.116@tcp11 [ 9494.434816] LNet: Removed LNI 192.168.203.116@tcp12 [ 9494.437123] LNet: Removed LNI 192.168.203.116@tcp13 [ 9494.439454] LNet: Removed LNI 192.168.203.116@tcp14 [ 9494.441770] LNet: Removed LNI 192.168.203.116@tcp15 [ 9494.444041] LNet: Removed LNI 192.168.203.116@tcp16 [ 9494.446145] LNet: Removed LNI 192.168.203.116@tcp17 [ 9494.448224] LNet: Removed LNI 192.168.203.116@tcp18 [ 9494.450858] LNet: Removed LNI 192.168.203.116@tcp19 [ 9494.452942] LNet: Removed LNI 192.168.203.116@tcp20 [ 9494.455082] LNet: Removed LNI 192.168.203.116@tcp21 [ 9494.457028] LNet: Removed LNI 192.168.203.116@tcp22 [ 9494.458881] LNet: Removed LNI 192.168.203.116@tcp23 [ 9494.460853] LNet: Removed LNI 192.168.203.116@tcp24 [ 9494.462862] LNet: Removed LNI 192.168.203.116@tcp25 [ 9494.464870] LNet: Removed LNI 192.168.203.116@tcp26 [ 9494.466742] LNet: Removed LNI 192.168.203.116@tcp27 [ 9494.468595] LNet: Removed LNI 192.168.203.116@tcp28 [ 9494.470442] LNet: Removed LNI 192.168.203.116@tcp29 [ 9494.472308] LNet: Removed LNI 192.168.203.116@tcp30 [ 9494.474029] LNet: Removed LNI 192.168.203.116@tcp31 [ 9494.475615] LNet: Removed LNI 192.168.203.116@tcp32 [ 9494.477337] LNet: Removed LNI 192.168.203.116@tcp33 [ 9494.478948] LNet: Removed LNI 192.168.203.116@tcp34 [ 9494.480527] LNet: Removed LNI 192.168.203.116@tcp35 [ 9494.482114] LNet: Removed LNI 192.168.203.116@tcp36 [ 9494.483623] LNet: Removed LNI 192.168.203.116@tcp37 [ 9494.485173] LNet: Removed LNI 192.168.203.116@tcp38 [ 9494.486691] LNet: Removed LNI 192.168.203.116@tcp39 [ 9494.488498] LNet: Removed LNI 192.168.203.116@tcp40 [ 9494.491227] LNet: Removed LNI 192.168.203.116@tcp41 [ 9494.494323] LNet: Removed LNI 192.168.203.116@tcp42 [ 9494.496899] LNet: Removed LNI 192.168.203.116@tcp43 [ 9494.499273] LNet: Removed LNI 192.168.203.116@tcp44 [ 9494.501545] LNet: Removed LNI 192.168.203.116@tcp45 [ 9494.503849] LNet: Removed LNI 192.168.203.116@tcp46 [ 9494.506332] LNet: Removed LNI 192.168.203.116@tcp47 [ 9494.508296] LNet: Removed LNI 192.168.203.116@tcp48 [ 9494.510349] LNet: Removed LNI 192.168.203.116@tcp49 [ 9494.518325] LNet: Removed LNI 192.168.203.116@tcp50 [ 9494.815912] Key type .llcrypt unregistered [ 9494.819236] Key type ._llcrypt unregistered [ 9503.406623] Key type ._llcrypt registered [ 9503.408122] Key type .llcrypt registered [ 9503.471578] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 9511.051738] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 9511.459286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9511.513657] alg: No test for adler32 (adler32-zlib) [ 9512.387573] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 9512.502171] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 9514.096178] Key type lgssc registered [ 9514.696443] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9527.879703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9527.885717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9529.000731] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9529.013600] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9529.054182] Lustre: lustre-MDT0000: new disk, initializing [ 9529.086845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9529.098187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9530.357619] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9534.607553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9534.648912] Lustre: 473664:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9534.674384] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9534.676272] Lustre: Skipped 1 previous similar message [ 9534.714490] Lustre: lustre-MDT0001: new disk, initializing [ 9534.747136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9534.757871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9534.762242] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9536.130588] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9539.414694] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9540.710588] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9543.238510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9543.353331] Lustre: *** cfs_fail_loc=90f, val=0*** [ 9543.397133] Lustre: lustre-OST0000: new disk, initializing [ 9543.399052] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9543.401745] Lustre: 475454:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9543.442066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9545.635619] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9545.861837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9545.865486] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9545.917227] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9549.281305] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9554.420681] Lustre: server umount lustre-OST0000 complete [ 9556.449626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9556.450553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9556.454449] Lustre: Skipped 1 previous similar message [ 9561.570253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9561.574688] Lustre: Skipped 2 previous similar messages [ 9565.707589] Lustre: server umount lustre-MDT0000 complete [ 9567.319316] LustreError: 473655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778932289 with bad export cookie 6997489022803172209 [ 9567.322416] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9567.322731] LustreError: 473655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9578.036946] Lustre: server umount lustre-MDT0001 complete [ 9582.340566] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 9583.783808] Key type lgssc unregistered [ 9583.974815] LNet: 477228:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9583.982502] LNetError: 477228:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9583.994659] LNet: Removed LNI 192.168.203.116@tcp [ 9584.423265] Key type .llcrypt unregistered [ 9584.424746] Key type ._llcrypt unregistered [ 9593.720822] Key type ._llcrypt registered [ 9593.722886] Key type .llcrypt registered [ 9593.776413] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 9594.248808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9594.278592] alg: No test for adler32 (adler32-zlib) [ 9595.129098] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 9595.228204] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 9596.832252] Key type lgssc registered [ 9597.219795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9600.809642] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9600.821496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9602.044580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9603.682870] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9607.627079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9607.835783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9609.617559] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9612.086092] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9613.792201] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9617.013269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9617.197884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9619.714385] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9622.370957] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9632.407927] loop0: detected capacity change from 0 to 204800000 [ 9632.493527] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9632.499458] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9632.505607] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9635.543434] loop0: detected capacity change from 0 to 204800000 [ 9635.597510] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9635.602144] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9635.606154] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9637.878149] loop0: detected capacity change from 0 to 204800000 [ 9640.559399] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9640.645357] Lustre: fs15246-OST0000: new disk, initializing [ 9640.649396] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 9640.655565] Lustre: 481716:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9640.692072] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9643.254397] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9648.754109] loop1: detected capacity change from 0 to 204800000 [ 9650.898280] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9650.934213] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 9650.955960] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 9650.994556] Lustre: fs15246-MDT0000: new disk, initializing [ 9651.046404] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9651.054935] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9651.098902] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9652.128717] Lustre: 482819:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@0000000078b2d8a9 x1865345733263744/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1778932390 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 9652.241753] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9661.219437] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9661.226112] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9661.240337] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9662.045089] Lustre: server umount fs15246-MDT0000 complete [ 9667.340201] Lustre: server umount fs15246-OST0000 complete [ 9672.673402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9672.680321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9672.689931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9674.722381] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9674.729967] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9674.742310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9676.261505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9676.265991] Lustre: Skipped 1 previous similar message [ 9677.928806] Lustre: server umount lustre-OST0000 complete [ 9679.841402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9679.845596] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9679.851674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9685.644463] Lustre: server umount lustre-MDT0000 complete [ 9686.497631] LustreError: 478459:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9686.511396] LustreError: 478459:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 9687.548689] LustreError: 478438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778932410 with bad export cookie 14958932967279622976 [ 9687.552760] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9687.557790] LustreError: 478438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9687.706678] Lustre: server umount lustre-MDT0001 complete [ 9691.710675] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 9692.873194] Key type lgssc unregistered [ 9693.035630] LNet: 485876:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9693.040798] LNetError: 485876:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9693.050579] LNet: Removed LNI 192.168.203.116@tcp [ 9693.521685] Key type .llcrypt unregistered [ 9693.524189] Key type ._llcrypt unregistered [ 9703.358565] Key type ._llcrypt registered [ 9703.360311] Key type .llcrypt registered [ 9703.423457] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 9703.709858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9703.749822] alg: No test for adler32 (adler32-zlib) [ 9704.613030] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 9704.703423] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 9706.288197] Key type lgssc registered [ 9706.880527] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9724.789421] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 9729.193316] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 9744.643418] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9744.654292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9745.762040] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9745.780298] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9745.831389] Lustre: lustre-MDT0000: new disk, initializing [ 9745.873892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9745.887046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9747.517478] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9752.687974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9752.747700] Lustre: 490259:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9752.793448] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9752.797371] Lustre: Skipped 1 previous similar message [ 9752.849599] Lustre: lustre-MDT0001: new disk, initializing [ 9752.895782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9752.917181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9752.923735] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9754.584566] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9758.301192] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9759.786674] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9762.885351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9763.017649] Lustre: lustre-OST0000: new disk, initializing [ 9763.020637] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9763.024864] Lustre: 491999:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9763.061486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9765.365709] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9768.987721] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9769.998093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9770.005564] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9770.056644] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9770.844150] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9770.955532] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9772.521316] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9772.594116] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9775.075887] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9775.077408] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9775.080756] LustreError: Skipped 1 previous similar message [ 9775.089797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9775.095396] Lustre: Skipped 1 previous similar message [ 9779.574753] Lustre: server umount lustre-OST0000 complete [ 9783.777573] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9783.782655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9783.790280] Lustre: Skipped 1 previous similar message [ 9783.793902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9785.313986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9785.315287] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9787.516879] Lustre: server umount lustre-MDT0000 complete [ 9789.346237] LustreError: 491037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778932512 with bad export cookie 17290989518888841500 [ 9789.352706] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9789.354148] LustreError: 491037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9789.493731] Lustre: server umount lustre-MDT0001 complete [ 9799.813342] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 9803.798210] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 9818.980742] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9537 sec ========== 07:55:40 (1778932540) [ 9819.742783] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 07:55:41 (1778932541) === [ 9821.316174] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 07:55:43 (1778932543) === [ 9839.751224] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 9841.354541] Key type lgssc unregistered [ 9841.537818] LNet: 498436:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9841.543152] LNetError: 498436:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9841.556710] LNet: Removed LNI 192.168.203.116@tcp [ 9842.062740] Key type .llcrypt unregistered [ 9842.064678] Key type ._llcrypt unregistered