[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 530710883 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003277] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005021] kvm-guest: setup PV IPIs [ 0.007849] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009016] pid_max: default: 32768 minimum: 301 [ 0.010153] LSM: Security Framework initializing [ 0.011070] Yama: becoming mindful. [ 0.012052] SELinux: Initializing. [ 0.013083] *** VALIDATE selinux *** [ 0.021730] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026691] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027194] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028132] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030134] *** VALIDATE tmpfs *** [ 0.031482] *** VALIDATE proc *** [ 0.033259] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035312] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036175] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038039] Spectre V2 : User space: Vulnerable [ 0.039014] Speculative Store Bypass: Vulnerable [ 0.041685] debug: unmapping init [mem 0xffffffffb5c59000-0xffffffffb5c60fff] [ 0.043237] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044770] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045035] ... version: 2 [ 0.046015] ... bit width: 48 [ 0.047013] ... generic registers: 4 [ 0.048013] ... value mask: 0000ffffffffffff [ 0.049015] ... max period: 00007fffffffffff [ 0.050018] ... fixed-purpose events: 3 [ 0.051018] ... event mask: 000000070000000f [ 0.052335] rcu: Hierarchical SRCU implementation. [ 0.054748] smp: Bringing up secondary CPUs ... [ 0.055639] x86: Booting SMP configuration: [ 0.056028] .... node #0, CPUs: #1 #2 #3 [ 0.065083] smp: Brought up 1 node, 4 CPUs [ 0.067017] smpboot: Max logical packages: 1 [ 0.068023] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.093314] node 0 deferred pages initialised in 23ms [ 0.097166] devtmpfs: initialized [ 0.098268] x86/mm: Memory block size: 128MB [ 0.100956] gcov: version magic: 0x41383552 [ 0.102197] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.103088] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.104273] pinctrl core: initialized pinctrl subsystem [ 0.105287] [ 0.105853] ************************************************************* [ 0.106014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.107015] ** ** [ 0.108014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.109018] ** ** [ 0.110019] ** This means that this kernel is built to expose internal ** [ 0.111016] ** IOMMU data structures, which may compromise security on ** [ 0.112017] ** your system. ** [ 0.113015] ** ** [ 0.114019] ** If you see this message and you are not debugging the ** [ 0.115014] ** kernel, report this immediately to your vendor! ** [ 0.116020] ** ** [ 0.117013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.118015] ************************************************************* [ 0.119761] NET: Registered protocol family 16 [ 0.120542] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.121069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.122071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.123427] cpuidle: using governor menu [ 0.126034] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.129495] PCI: Using configuration type 1 for base access [ 0.132164] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.144055] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.147059] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.151046] cryptd: max_cpu_qlen set to 1000 [ 0.154215] ACPI: Added _OSI(Module Device) [ 0.155012] ACPI: Added _OSI(Processor Device) [ 0.157015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.159016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.165042] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.171582] ACPI: Interpreter enabled [ 0.173069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.175015] ACPI: Using IOAPIC for interrupt routing [ 0.177302] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.181397] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.192792] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.195060] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.197020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.200082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.205532] acpiphp: Slot [2] registered [ 0.207134] acpiphp: Slot [5] registered [ 0.209150] acpiphp: Slot [6] registered [ 0.210105] acpiphp: Slot [7] registered [ 0.211180] acpiphp: Slot [8] registered [ 0.213138] acpiphp: Slot [9] registered [ 0.214135] acpiphp: Slot [10] registered [ 0.215173] acpiphp: Slot [3] registered [ 0.216058] acpiphp: Slot [4] registered [ 0.218099] acpiphp: Slot [11] registered [ 0.219000] acpiphp: Slot [12] registered [ 0.219000] acpiphp: Slot [13] registered [ 0.220111] acpiphp: Slot [14] registered [ 0.221101] acpiphp: Slot [15] registered [ 0.223109] acpiphp: Slot [16] registered [ 0.224169] acpiphp: Slot [17] registered [ 0.226121] acpiphp: Slot [18] registered [ 0.227104] acpiphp: Slot [19] registered [ 0.229134] acpiphp: Slot [20] registered [ 0.230108] acpiphp: Slot [21] registered [ 0.231128] acpiphp: Slot [22] registered [ 0.233139] acpiphp: Slot [23] registered [ 0.235113] acpiphp: Slot [24] registered [ 0.236095] acpiphp: Slot [25] registered [ 0.238144] acpiphp: Slot [26] registered [ 0.239093] acpiphp: Slot [27] registered [ 0.241099] acpiphp: Slot [28] registered [ 0.242095] acpiphp: Slot [29] registered [ 0.244104] acpiphp: Slot [30] registered [ 0.245094] acpiphp: Slot [31] registered [ 0.246063] PCI host bridge to bus 0000:00 [ 0.247022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.249019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.252024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.254038] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.256035] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.258031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.260187] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.263245] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.267364] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.277016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.283063] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.286021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.289023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.292024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.294408] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.295628] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.297051] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.299765] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.304018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.315732] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.319014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.323139] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.331022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.339020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.374019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.397098] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.406031] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.417018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.438020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.450716] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.457020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.469017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.494017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.504300] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.511018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.522020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.544014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.554016] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.563015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.574025] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.593018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.607233] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.616017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.626019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.642023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.655257] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.656273] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.658309] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.659214] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.661235] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.666092] iommu: Default domain type: Passthrough [ 0.668350] SCSI subsystem initialized [ 0.669123] ACPI: bus type USB registered [ 0.670083] usbcore: registered new interface driver usbfs [ 0.671072] usbcore: registered new interface driver hub [ 0.673073] usbcore: registered new device driver usb [ 0.674106] pps_core: LinuxPPS API ver. 1 registered [ 0.675008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.677051] PTP clock support registered [ 0.679063] EDAC MC: Ver: 3.0.0 [ 0.681124] PCI: Using ACPI for IRQ routing [ 0.682541] NetLabel: Initializing [ 0.683008] NetLabel: domain hash size = 128 [ 0.684007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.685067] NetLabel: unlabeled traffic allowed by default [ 0.688119] vgaarb: loaded [ 0.689223] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.690008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.694948] clocksource: Switched to clocksource kvm-clock [ 0.796231] VFS: Disk quotas dquot_6.6.0 [ 0.797453] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.799130] *** VALIDATE ramfs *** [ 0.800484] *** VALIDATE hugetlbfs *** [ 0.802345] pnp: PnP ACPI init [ 0.804701] pnp: PnP ACPI: found 6 devices [ 0.820825] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.823673] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.825687] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.827146] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.828584] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.830271] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.831969] NET: Registered protocol family 2 [ 0.833558] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.837129] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.839926] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.844636] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.847548] TCP: Hash tables configured (established 65536 bind 65536) [ 0.850275] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.852577] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.854914] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.857219] NET: Registered protocol family 1 [ 0.859749] RPC: Registered named UNIX socket transport module. [ 0.861544] RPC: Registered udp transport module. [ 0.862832] RPC: Registered tcp transport module. [ 0.864148] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.865812] NET: Registered protocol family 44 [ 0.866950] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.868569] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.870029] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.871683] PCI: CLS 0 bytes, default 64 [ 0.873171] Unpacking initramfs... [ 2.216355] debug: unmapping init [mem 0xffff91bb7cc54000-0xffff91bb7ffbffff] [ 2.220339] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.222513] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.225573] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.763893] Initialise system trusted keyrings [ 2.765618] Key type blacklist registered [ 2.767371] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.775963] zbud: loaded [ 2.778177] *** VALIDATE nfs *** [ 2.779232] *** VALIDATE nfs4 *** [ 2.780678] pstore: using deflate compression [ 2.783604] Platform Keyring initialized [ 2.885627] NET: Registered protocol family 38 [ 2.887524] Key type asymmetric registered [ 2.889539] Asymmetric key parser 'x509' registered [ 2.891451] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.894303] io scheduler mq-deadline registered [ 2.896196] io scheduler kyber registered [ 2.897834] io scheduler bfq registered [ 2.900091] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.903202] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.905753] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.907769] ACPI: Power Button [PWRF] [ 2.913831] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.920668] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.969113] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.978388] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.997818] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.027422] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.060075] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.065400] Non-volatile memory driver v1.3 [ 3.067017] Linux agpgart interface v0.103 [ 3.102066] virtio_blk virtio1: [vda] 134720 512-byte logical blocks (69.0 MB/65.8 MiB) [ 3.105026] vda: detected capacity change from 0 to 68976640 [ 3.119572] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.122369] vdb: detected capacity change from 0 to 1073741824 [ 3.136616] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.140044] vdc: detected capacity change from 0 to 2621440000 [ 3.159610] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.162586] vdd: detected capacity change from 0 to 2621440000 [ 3.175361] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.177361] vde: detected capacity change from 0 to 4294967296 [ 3.190432] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.193681] vdf: detected capacity change from 0 to 4294967296 [ 3.201908] libphy: Fixed MDIO Bus: probed [ 3.211912] usbcore: registered new interface driver usbserial_generic [ 3.214073] usbserial: USB Serial support registered for generic [ 3.215522] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.219382] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.220509] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.222522] mousedev: PS/2 mouse device common for all mice [ 3.225579] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.229422] rtc_cmos 00:05: RTC can wake from S4 [ 3.233282] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.233456] rtc_cmos 00:05: registered as rtc0 [ 3.238206] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.239678] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.240486] intel_pstate: CPU model not supported [ 3.246258] hid: raw HID events driver (C) Jiri Kosina [ 3.247698] usbcore: registered new interface driver usbhid [ 3.249116] usbhid: USB HID core driver [ 3.250029] drop_monitor: Initializing network drop monitor service [ 3.251469] Initializing XFRM netlink socket [ 3.252701] NET: Registered protocol family 10 [ 3.255218] Segment Routing with IPv6 [ 3.256845] NET: Registered protocol family 17 [ 3.259032] mpls_gso: MPLS GSO support [ 3.264328] RAS: Correctable Errors collector initialized. [ 3.265847] AVX version of gcm_enc/dec engaged. [ 3.267465] AES CTR mode by8 optimization enabled [ 3.352586] sched_clock: Marking stable (3352502465, 0)->(4357565947, -1005063482) [ 3.356453] registered taskstats version 1 [ 3.358491] Loading compiled-in X.509 certificates [ 3.360610] zswap: loaded using pool lzo/zbud [ 3.384960] Key type big_key registered [ 3.397139] Key type encrypted registered [ 3.398780] ima: No TPM chip found, activating TPM-bypass! [ 3.401178] ima: Allocated hash algorithm: sha1 [ 3.402802] ima: No architecture policies found [ 3.404533] evm: Initialising EVM extended attributes: [ 3.406478] evm: security.selinux [ 3.407714] evm: security.ima [ 3.408862] evm: security.capability [ 3.410266] evm: HMAC attrs: 0x1 [ 3.412525] rtc_cmos 00:05: setting system clock to 2026-03-02 07:14:23 UTC (1772435663) [ 3.418795] debug: unmapping init [mem 0xffffffffb6c03000-0xffffffffb6dfffff] [ 3.421785] debug: unmapping init [mem 0xffffffffb5982000-0xffffffffb5c58fff] [ 3.430120] Write protecting the kernel read-only data: 28672k [ 3.433886] debug: unmapping init [mem 0xffffffffb4003000-0xffffffffb41fffff] [ 3.436919] debug: unmapping init [mem 0xffffffffb4914000-0xffffffffb49fffff] [ 3.471187] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.479242] systemd[1]: Detected virtualization kvm. [ 3.481154] systemd[1]: Detected architecture x86-64. [ 3.483066] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.509274] systemd[1]: No hostname configured. [ 3.510407] systemd[1]: Set hostname to . [ 3.511817] random: systemd: uninitialized urandom read (16 bytes read) [ 3.514357] systemd[1]: Initializing machine ID from random generator. [ 3.627632] random: systemd: uninitialized urandom read (16 bytes read) [ 3.629908] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.634349] random: systemd: uninitialized urandom read (16 bytes read) [ 3.636768] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.640596] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.228446] device-mapper: uevent: version 1.0.3 [ 4.230774] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.909863] virtio_net virtio0 ens2: renamed from eth0 [ 4.933418] random: fast init done [ 4.973636] scsi host0: ata_piix [ 5.044001] scsi host1: ata_piix [ 5.045690] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.048250] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.080254] dracut-initqueue[574]: RTNETLINK answers: File exists [ 9.914910] random: crng init done [ 9.916680] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.407380] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.504378] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.768369] SELinux: Disabled at runtime. [ 11.825283] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.830376] systemd[1]: Detected virtualization kvm. [ 11.831376] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.280390] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.283731] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.288977] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.293710] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.297486] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.304549] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.307743] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ 12.365136] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.783275] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.091521] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.112985] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.240216] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.261156] EDAC sbridge: Ver: 1.1.2 [ 14.955834] Key type dns_resolver registered [ 15.271342] NFS: Registering the id_resolver key type [ 15.272774] Key type id_resolver registered [ 15.273759] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg141-server login: [ 40.567055] hrtimer: interrupt took 2207052 ns [ 63.543503] libcfs: loading out-of-tree module taints kernel. [ 63.668308] Key type ._llcrypt registered [ 63.670327] Key type .llcrypt registered [ 63.831327] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 83.833822] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 85.597991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 85.623282] alg: No test for adler32 (adler32-zlib) [ 86.940330] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 88.058619] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 89.895177] Key type lgssc registered [ 91.584829] Lustre: Echo OBD driver; http://www.lustre.org/ [ 107.981809] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 157.103472] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 170.606614] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 170.666865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 172.005689] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 172.030568] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 172.138808] Lustre: lustre-MDT0000: new disk, initializing [ 172.283116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 172.307815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 176.728938] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 189.645058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 189.746043] Lustre: 6496:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 189.786626] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 189.795247] Lustre: Skipped 1 previous similar message [ 189.884617] Lustre: lustre-MDT0001: new disk, initializing [ 189.948506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 189.992240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 190.012265] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 194.825121] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 200.143227] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 209.672761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 209.944457] Lustre: lustre-OST0000: new disk, initializing [ 209.949126] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 210.018177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 211.063600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 211.069987] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 211.195740] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 216.421606] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.156304] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 229.263906] Lustre: lustre-OST0001: new disk, initializing [ 229.267680] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 229.323567] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 236.073791] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 238.648354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 238.660045] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 238.721609] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 247.804694] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 253.349188] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 264.984532] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing check_logdir /tmp/testlogs/ [ 269.470879] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing yml_node [ 273.717523] Lustre: DEBUG MARKER: Client: 2.17.50.193 [ 276.129107] Lustre: DEBUG MARKER: MDS: 2.17.50.193 [ 278.609617] Lustre: DEBUG MARKER: OSS: 2.17.50.193 [ 280.159546] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Mar 2 02:18:58 EST 2026 [ 297.778258] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 299.333619] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 315.364516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 315.376374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 315.394023] Lustre: Skipped 3 previous similar messages [ 315.402951] Lustre: Skipped 3 previous similar messages [ 320.702732] Lustre: server umount lustre-MDT0000 complete [ 323.698755] LustreError: 6488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772435983 with bad export cookie 17432271177630192698 [ 323.706421] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 323.714223] LustreError: 6488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 325.600508] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 325.600938] LustreError: 8405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 325.613831] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 325.634125] LustreError: 8405:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 325.642804] Lustre: Skipped 1 previous similar message [ 328.608145] LustreError: 8405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 328.608582] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 328.628413] LustreError: 8405:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 328.642083] Lustre: Skipped 1 previous similar message [ 333.728447] LustreError: 8405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 333.729454] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 333.747633] LustreError: 8405:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 333.753362] Lustre: Skipped 1 previous similar message [ 338.399192] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 338.709091] Lustre: server umount lustre-MDT0001 complete [ 342.630422] Lustre: server umount lustre-OST0000 complete [ 346.543421] Lustre: server umount lustre-OST0001 complete [ 354.157701] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 363.677297] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 395.900089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 396.140267] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 396.174946] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 396.253292] Lustre: lustre-MDT0000: new disk, initializing [ 396.345886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 396.367512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 400.630821] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 410.294156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 410.423815] Lustre: 14968:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 410.489973] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 410.495319] Lustre: Skipped 1 previous similar message [ 410.562509] Lustre: lustre-MDT0001: new disk, initializing [ 410.657222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 410.691782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 410.701299] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 414.855749] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 421.891814] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 426.097883] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 432.582410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 432.860760] Lustre: lustre-OST0000: new disk, initializing [ 432.863706] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 432.942247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 434.717208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 434.730291] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 434.878818] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 438.426251] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 445.651396] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 451.588733] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 451.788910] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 455.566566] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 455.719712] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 460.771740] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 460.780591] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 460.785392] LustreError: Skipped 1 previous similar message [ 460.789109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 460.803265] Lustre: Skipped 2 previous similar messages [ 463.463167] Lustre: server umount lustre-OST0000 complete [ 466.914713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 466.929376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 471.011889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 471.013680] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 471.017831] Lustre: Skipped 2 previous similar messages [ 472.593578] Lustre: server umount lustre-MDT0000 complete [ 475.547939] LustreError: 15748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772436135 with bad export cookie 17432271177630194231 [ 475.551884] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 475.556213] LustreError: 15748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 475.748343] Lustre: server umount lustre-MDT0001 complete [ 478.710526] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 02:22:17 (1772436137) [ 480.766714] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 482.148965] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 02:22:20 (1772436140) [ 488.407261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 488.981619] LustreError: 18971:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 489.053982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 492.656653] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 494.051787] LustreError: 18972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 494.078756] LustreError: 18972:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 499.634700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 503.484351] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 507.796722] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 510.271675] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 515.157607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 520.191387] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 525.038393] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 544.224403] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 544.228636] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 544.234027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 544.238581] Lustre: Skipped 1 previous similar message [ 544.247618] Lustre: Skipped 2 previous similar messages [ 547.505798] Lustre: server umount lustre-OST0000 complete [ 551.392331] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 555.951389] Lustre: server umount lustre-MDT0000 complete [ 558.266273] LustreError: 18952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772436218 with bad export cookie 17432271177630195106 [ 558.269721] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 558.287335] LustreError: 18952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 558.561511] Lustre: server umount lustre-MDT0001 complete [ 566.888186] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 569.214779] Key type lgssc unregistered [ 569.431859] LNet: 22171:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 569.438224] LNetError: 22171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 569.450248] LNet: Removed LNI 192.168.201.141@tcp [ 570.119179] Key type .llcrypt unregistered [ 570.122932] Key type ._llcrypt unregistered [ 588.031074] Key type ._llcrypt registered [ 588.033034] Key type .llcrypt registered [ 588.116476] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 588.765864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 588.856899] alg: No test for adler32 (adler32-zlib) [ 589.821715] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 589.988484] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 591.631171] Key type lgssc registered [ 592.415824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 599.421777] loop: module loaded [ 599.485105] loop0: detected capacity change from 0 to 204800000 [ 599.673885] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 599.683457] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 599.898585] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 605.121746] loop0: detected capacity change from 0 to 204800000 [ 605.239497] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 605.254206] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 605.271174] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 609.753727] loop0: detected capacity change from 0 to 204800000 [ 614.352385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 614.379215] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 615.679327] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 615.774137] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 615.892889] Lustre: lustre-MDT0000: new disk, initializing [ 616.017545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 616.040171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 619.732994] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 626.600425] loop1: detected capacity change from 0 to 204800000 [ 630.897125] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 631.109615] Lustre: lustre-OST07c6: new disk, initializing [ 631.113176] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 631.117891] Lustre: Skipped 1 previous similar message [ 631.165892] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 631.593288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 631.601094] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 631.630243] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 635.253271] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 638.843912] Lustre: 24308:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 638.852373] Lustre: 24308:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 638.857199] Lustre: 24308:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 638.864424] Lustre: 24308:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 638.868316] Lustre: 24308:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 638.872135] Lustre: 24308:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 661.467831] Lustre: 24307:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 661.478815] Lustre: 24307:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 661.483797] Lustre: 24307:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 661.487583] Lustre: 24307:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 661.493354] Lustre: 24307:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 661.495904] Lustre: 24307:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 665.803792] Lustre: 24308:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 665.810606] Lustre: 24308:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 1 previous similar message [ 665.815363] Lustre: 24308:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 665.818952] Lustre: 24308:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 665.823056] Lustre: 24308:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 665.827389] Lustre: 24308:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 665.830754] Lustre: 24308:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 665.833450] Lustre: 24308:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 665.835715] Lustre: 24308:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 665.839816] Lustre: 24308:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 665.843969] Lustre: 24308:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 665.846414] Lustre: 24308:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 669.002332] Lustre: 25760:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 669.014205] Lustre: 25760:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 1 previous similar message [ 669.026129] Lustre: 25760:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 669.038076] Lustre: 25760:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 669.041438] Lustre: 25760:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 669.045669] Lustre: 25760:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 669.049891] Lustre: 25760:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/1 [ 669.054395] Lustre: 25760:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 669.057935] Lustre: 25760:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 669.061704] Lustre: 25760:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 669.066074] Lustre: 25760:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 669.069794] Lustre: 25760:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 673.663578] Lustre: 24306:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 673.677543] Lustre: 24306:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 1 previous similar message [ 673.691189] Lustre: 24306:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 673.699961] Lustre: 24306:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 673.705448] Lustre: 24306:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 673.711144] Lustre: 24306:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 673.716845] Lustre: 24306:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/1 [ 673.722181] Lustre: 24306:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 673.726852] Lustre: 24306:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 673.742671] Lustre: 24306:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 673.750449] Lustre: 24306:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 673.756266] Lustre: 24306:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 685.038712] Lustre: 24308:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 685.043456] Lustre: 24308:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 5 previous similar messages [ 685.049612] Lustre: 24308:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 685.055683] Lustre: 24308:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 685.060527] Lustre: 24308:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 685.065689] Lustre: 24308:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 685.071200] Lustre: 24308:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/1 [ 685.076411] Lustre: 24308:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 685.081734] Lustre: 24308:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 685.086193] Lustre: 24308:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 685.093779] Lustre: 24308:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 685.100560] Lustre: 24308:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 734.184086] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 734.207887] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 734.208411] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 734.384790] Lustre: server umount lustre-OST07c6 complete [ 741.488465] Lustre: server umount lustre-MDT0000 complete [ 753.550704] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 755.752743] Key type lgssc unregistered [ 755.955477] LNet: 27825:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 755.962662] LNetError: 27825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 755.976496] LNet: Removed LNI 192.168.201.141@tcp [ 756.469435] Key type .llcrypt unregistered [ 756.471284] Key type ._llcrypt unregistered [ 769.368221] Key type ._llcrypt registered [ 769.370846] Key type .llcrypt registered [ 769.463583] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 770.088060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 770.205245] alg: No test for adler32 (adler32-zlib) [ 771.174391] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 771.339110] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 772.991337] Key type lgssc registered [ 773.846860] Lustre: Echo OBD driver; http://www.lustre.org/ [ 779.494712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 779.503845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 780.851275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 783.665664] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 789.313170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 789.600280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 792.533316] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 796.334569] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 798.498466] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 803.326373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 803.575486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 805.629759] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 807.696324] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 811.958415] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 830.433486] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 830.441971] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 830.445056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 830.455917] Lustre: Skipped 1 previous similar message [ 830.470985] Lustre: Skipped 1 previous similar message [ 832.595855] Lustre: server umount lustre-OST0000 complete [ 835.554086] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 835.565924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 835.566214] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 840.673708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 840.679386] Lustre: Skipped 2 previous similar messages [ 841.812160] Lustre: server umount lustre-MDT0000 complete [ 844.517133] LustreError: 29037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772436504 with bad export cookie 8613545361818315508 [ 844.521402] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 844.528420] LustreError: 29037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 844.851474] Lustre: server umount lustre-MDT0001 complete [ 851.784223] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 853.876830] Key type lgssc unregistered [ 854.188472] LNet: 32435:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 854.195845] LNetError: 32435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 854.214544] LNet: Removed LNI 192.168.201.141@tcp [ 854.852961] Key type .llcrypt unregistered [ 854.855428] Key type ._llcrypt unregistered [ 867.347285] Key type ._llcrypt registered [ 867.349569] Key type .llcrypt registered [ 867.440399] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 868.174343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 868.246552] alg: No test for adler32 (adler32-zlib) [ 869.219456] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 869.374935] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 871.015466] Key type lgssc registered [ 871.814667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 877.653386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 877.668905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 879.030338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 881.987896] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 888.050161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 888.288693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 891.193598] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 895.256045] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 897.992496] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 902.678278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 902.918799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 903.991788] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 907.860105] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 912.266428] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 923.619920] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 923.620332] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 923.668708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 923.681428] Lustre: Skipped 1 previous similar message [ 927.471535] Lustre: server umount lustre-OST0000 complete [ 933.863483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 933.867908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 933.880426] Lustre: Skipped 2 previous similar messages [ 933.893024] Lustre: Skipped 1 previous similar message [ 936.752774] Lustre: server umount lustre-MDT0000 complete [ 938.976233] LustreError: 34377:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 938.995621] LustreError: 34377:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 939.868335] LustreError: 33647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772436599 with bad export cookie 427666338741740162 [ 939.880206] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 939.882342] LustreError: 33647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 940.106366] Lustre: server umount lustre-MDT0001 complete [ 949.392469] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 951.287138] Key type lgssc unregistered [ 951.473684] LNet: 36847:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 951.482107] LNetError: 36847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 951.493891] LNet: Removed LNI 192.168.201.141@tcp [ 951.988257] Key type .llcrypt unregistered [ 951.989815] Key type ._llcrypt unregistered [ 969.261884] Key type ._llcrypt registered [ 969.264935] Key type .llcrypt registered [ 969.344575] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 970.121480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 970.144442] alg: No test for adler32 (adler32-zlib) [ 971.185799] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 971.410602] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 973.047316] Key type lgssc registered [ 973.924963] Lustre: Echo OBD driver; http://www.lustre.org/ [ 979.123079] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 979.139330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 980.471312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 983.601326] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 989.764908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 990.128298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 993.417590] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 997.391841] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 999.843546] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1004.881803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1005.200681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1006.259302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1009.416130] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1013.134486] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1020.730938] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.41@tcp (stopping) [ 1020.840043] Lustre: server umount lustre-MDT0000 complete [ 1020.901039] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1020.902552] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1020.917440] LustreError: 38227:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1020.932792] LustreError: 38227:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1023.511582] LustreError: 38208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772436683 with bad export cookie 15505630386025264724 [ 1023.517166] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1023.521241] LustreError: 38208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1025.055299] LustreError: 38937:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000015b31843 x1858534512812032/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1025.832370] LustreError: 39335:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1025.848432] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.41@tcp (stopping) [ 1026.019168] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1026.028021] Lustre: Skipped 2 previous similar messages [ 1028.063781] LustreError: 38222:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1028.063905] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1028.074918] LustreError: 38222:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1028.088339] Lustre: Skipped 1 previous similar message [ 1029.768248] Lustre: server umount lustre-MDT0001 complete [ 1048.741830] Lustre: server umount lustre-OST0000 complete [ 1059.852341] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1061.904223] Key type lgssc unregistered [ 1062.096123] LNet: 41729:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1062.101761] LNetError: 41729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1062.117182] LNet: Removed LNI 192.168.201.141@tcp [ 1062.700383] Key type .llcrypt unregistered [ 1062.705525] Key type ._llcrypt unregistered [ 1078.384458] Key type ._llcrypt registered [ 1078.386481] Key type .llcrypt registered [ 1078.448457] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1079.264504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1079.278061] alg: No test for adler32 (adler32-zlib) [ 1080.246843] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 1080.403103] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 1082.047187] Key type lgssc registered [ 1082.711919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1088.050960] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1088.058533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1089.352701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1092.077637] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1097.004659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1097.225439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1099.837119] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1103.882980] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1106.577736] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1111.476245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1111.689184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1114.468342] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1114.793409] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1117.551289] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1127.392300] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1127.394449] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1127.413182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1128.568474] Lustre: server umount lustre-OST0000 complete [ 1143.265395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1143.277849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1143.300573] Lustre: Skipped 1 previous similar message [ 1143.307477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1143.313028] Lustre: Skipped 1 previous similar message [ 1147.426607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1147.428955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1152.480801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1152.483064] Lustre: Skipped 1 previous similar message [ 1157.087349] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1157.195222] Lustre: server umount lustre-MDT0000 complete [ 1157.602133] LustreError: 43102:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1157.613930] LustreError: 43102:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1159.635306] LustreError: 43087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772436819 with bad export cookie 8556027383079380259 [ 1159.640982] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1159.643103] LustreError: 43087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1166.023742] Lustre: server umount lustre-MDT0001 complete [ 1171.624616] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1173.189304] Key type lgssc unregistered [ 1173.381053] LNet: 46456:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1173.386149] LNetError: 46456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1173.402875] LNet: Removed LNI 192.168.201.141@tcp [ 1173.864517] Key type .llcrypt unregistered [ 1173.866353] Key type ._llcrypt unregistered [ 1187.713515] Key type ._llcrypt registered [ 1187.714779] Key type .llcrypt registered [ 1187.789360] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1188.360583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1188.400048] alg: No test for adler32 (adler32-zlib) [ 1189.375360] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 1189.516398] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 1191.143184] Key type lgssc registered [ 1191.927100] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1196.954377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1196.966265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1198.265634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1200.917902] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1206.770805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1207.056834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1210.326917] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1214.427765] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1217.308531] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1222.203225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1222.511156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1223.533755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1226.868132] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1230.769789] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1236.345961] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1238.877330] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1250.201806] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-03-02 2h35m08s [ 1251.060562] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1253.346834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1253.352613] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1253.354340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1253.357464] Lustre: Skipped 2 previous similar messages [ 1254.368604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1254.377689] Lustre: Skipped 3 previous similar messages [ 1258.589667] Lustre: server umount lustre-MDT0000 complete [ 1259.816866] LustreError: 47827:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1259.826756] LustreError: 47827:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1260.515461] LustreError: 47833:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1260.528614] LustreError: 47833:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1260.966771] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1265.508817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1265.608994] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1265.804541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1265.854748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1266.797835] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1268.369412] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1270.502988] Lustre: DEBUG MARKER: Wait for df (36037) ... [ 1271.507675] Lustre: DEBUG MARKER: done [ 1273.832918] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1273.838676] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1273.852355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1273.855696] Lustre: Skipped 3 previous similar messages [ 1275.881722] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1275.887458] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1275.899198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1279.791949] Lustre: server umount lustre-OST0000 complete [ 1282.531215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1282.532961] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1282.533686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1282.533691] Lustre: Skipped 2 previous similar messages [ 1282.536837] Lustre: Skipped 1 previous similar message [ 1294.818847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1294.825689] Lustre: Skipped 6 previous similar messages [ 1296.351176] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1296.509483] Lustre: server umount lustre-MDT0000 complete [ 1299.209948] LustreError: 47814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772436959 with bad export cookie 16309117856160825744 [ 1299.216426] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1299.219132] LustreError: 47814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1299.473533] Lustre: server umount lustre-MDT0001 complete [ 1306.625916] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1308.633496] Key type lgssc unregistered [ 1308.867833] LNet: 52383:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1308.873489] LNetError: 52383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1309.925370] LNet: Removed LNI 192.168.201.141@tcp [ 1310.583192] Key type .llcrypt unregistered [ 1310.585670] Key type ._llcrypt unregistered [ 1337.195471] Key type ._llcrypt registered [ 1337.196851] Key type .llcrypt registered [ 1337.258894] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1338.192787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1338.213046] alg: No test for adler32 (adler32-zlib) [ 1339.192570] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 1339.386215] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 1341.047164] Key type lgssc registered [ 1341.809822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1347.001777] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1347.019200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1348.182668] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1348.202824] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1348.310058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1351.449897] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1356.981475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1357.036874] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1357.056605] Lustre: 54552:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1357.198863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1359.851330] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1363.735761] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1366.541933] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1371.355199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1371.450834] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1371.610749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1376.045551] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1380.344419] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1380.899859] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1386.327362] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-03-02 2h37m25s [ 1387.343395] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1391.887570] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1393.617427] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1393.644375] LustreError: 54559:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1413.953125] Lustre: lustre-MDT0000: Client be8fa7bd-bb19-47d6-bb54-5538975e4df5 (at 192.168.201.41@tcp) reconnecting [ 1433.720090] LustreError: 54559:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1435.373864] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1437.030855] Lustre: DEBUG MARKER: done [ 1442.275789] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1442.282412] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1442.288896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1442.294744] LustreError: Skipped 1 previous similar message [ 1442.302595] Lustre: Skipped 1 previous similar message [ 1446.629108] Lustre: server umount lustre-OST0000 complete [ 1449.439937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1449.442538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1449.447149] Lustre: Skipped 1 previous similar message [ 1449.449210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1450.467226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1450.473137] Lustre: Skipped 3 previous similar messages [ 1452.514898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1456.609633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1456.613860] Lustre: Skipped 3 previous similar messages [ 1463.263505] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1463.370630] Lustre: server umount lustre-MDT0000 complete [ 1465.593675] LustreError: 54546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772437125 with bad export cookie 17174773349149557296 [ 1465.600087] LustreError: 54546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1465.600547] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1465.769600] Lustre: server umount lustre-MDT0001 complete [ 1471.612158] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1473.370504] Key type lgssc unregistered [ 1473.550910] LNet: 58521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1473.555592] LNetError: 58521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1473.564575] LNet: Removed LNI 192.168.201.141@tcp [ 1474.097649] Key type .llcrypt unregistered [ 1474.099282] Key type ._llcrypt unregistered [ 1500.857702] Key type ._llcrypt registered [ 1500.860324] Key type .llcrypt registered [ 1500.943691] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1501.872047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1501.896337] alg: No test for adler32 (adler32-zlib) [ 1502.882786] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 1503.048066] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 1504.655202] Key type lgssc registered [ 1505.348561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1507.371251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1507.381126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1508.538134] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1508.550766] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1508.626351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1509.537785] Lustre: Failing over lustre-MDT0000 [ 1509.800579] Lustre: server umount lustre-MDT0000 complete [ 1515.418727] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 02:39:32 (1772437172) [ 1519.646185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1519.948720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1523.246990] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1528.839081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1528.907165] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1528.931940] Lustre: 61282:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1529.088030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1532.255584] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1536.209386] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1538.957984] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1543.414437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1543.497764] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1543.647317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1547.592920] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1551.500929] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1552.940057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1561.121477] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1565.152275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1565.157887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1565.165304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1568.228565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1568.228574] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1568.250900] Lustre: Skipped 1 previous similar message [ 1571.007072] Lustre: server umount lustre-MDT0000 complete [ 1573.232863] LustreError: 61275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772437233 with bad export cookie 6337209756558112111 [ 1573.235193] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1573.239847] LustreError: 61275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1573.354062] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1573.480955] Lustre: server umount lustre-MDT0001 complete [ 1575.661075] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1579.897643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1585.408229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1598.087894] LustreError: 62903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1598.097710] LustreError: 62903:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1598.169931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1598.254158] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1601.653618] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1602.211393] LustreError: 62904:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1607.654795] LustreError: 62903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1607.725863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1608.030773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1611.440395] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1613.160756] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1615.334928] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1618.372028] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1623.523436] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1623.524982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1623.530789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1623.538497] Lustre: Skipped 3 previous similar messages [ 1625.377726] Lustre: server umount lustre-MDT0000 complete [ 1628.071599] LustreError: 62910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772437288 with bad export cookie 6337209756558114393 [ 1628.073412] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1628.077905] LustreError: 62910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1628.379169] Lustre: server umount lustre-MDT0001 complete [ 1633.103719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1640.483360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1645.791203] Lustre: 60207:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772437289/real 1772437289] req@00000000b1da89f3 x1858535070687360/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772437305 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1645.824062] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1645.830107] Lustre: Skipped 1 previous similar message [ 1649.119166] Lustre: 60209:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772437293/real 1772437293] req@00000000b28f1696 x1858535070687488/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772437309 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1650.145092] Lustre: 60209:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772437294/real 1772437294] req@00000000a5ccde5b x1858535070687616/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772437310 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1653.229831] LustreError: 62904:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1653.297160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1653.365618] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1654.303163] Lustre: 60206:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772437298/real 1772437298] req@00000000ca5b6e32 x1858535070687872/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772437314 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1654.315535] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1656.295906] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1660.387515] LustreError: 62903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1660.395526] LustreError: 62903:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1660.425067] Lustre: 60206:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772437304/real 1772437304] req@000000002a6d2a67 x1858535070688000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772437320 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1662.327096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1665.646378] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1668.091574] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1669.684211] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1672.215569] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1678.308561] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1678.312113] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1678.321366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1678.338964] Lustre: Skipped 5 previous similar messages [ 1682.771396] Lustre: server umount lustre-MDT0000 complete [ 1683.425175] LustreError: 67269:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1683.442706] LustreError: 67269:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1685.459602] LustreError: 61276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772437345 with bad export cookie 6337209756558115765 [ 1685.472256] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1685.735855] Lustre: server umount lustre-MDT0001 complete [ 1690.216681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1692.549768] Lustre: DEBUG MARKER: files compared the same [ 1704.417057] Lustre: server umount lustre-OST0000 complete [ 1716.171542] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1718.252685] Key type lgssc unregistered [ 1718.437577] LNet: 70200:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1718.441103] LNetError: 70200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1719.464143] LNet: Removed LNI 192.168.201.141@tcp [ 1720.170102] Key type .llcrypt unregistered [ 1720.171639] Key type ._llcrypt unregistered [ 1735.974391] Key type ._llcrypt registered [ 1735.982675] Key type .llcrypt registered [ 1736.025277] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1736.972443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1737.010426] alg: No test for adler32 (adler32-zlib) [ 1737.974856] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 1738.114426] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 1739.727278] Key type lgssc registered [ 1740.591918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1746.144481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1746.169141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1747.549786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1750.193868] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug +malloc all [ 1756.489206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1756.772956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1760.077846] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug +malloc all [ 1763.910424] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1766.453955] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1771.129467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1771.461578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1772.526805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1775.922225] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug +malloc all [ 1780.002714] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1786.852465] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1786.852814] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1786.856592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1786.861807] Lustre: Skipped 1 previous similar message [ 1786.868502] Lustre: Skipped 1 previous similar message [ 1788.442826] Lustre: server umount lustre-OST0000 complete [ 1790.473837] Lustre: server umount lustre-MDT0000 complete [ 1791.968726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1791.970638] LustreError: 73628:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1791.975960] Lustre: Skipped 1 previous similar message [ 1791.983684] LustreError: 73628:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1792.604453] LustreError: 71558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772437452 with bad export cookie 1455868437137218101 [ 1792.616384] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1792.751399] Lustre: server umount lustre-MDT0001 complete [ 1799.083458] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1800.725384] Key type lgssc unregistered [ 1800.921637] LNet: 74752:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1800.930135] LNetError: 74752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1800.943485] LNet: Removed LNI 192.168.201.141@tcp [ 1801.401482] Key type .llcrypt unregistered [ 1801.403543] Key type ._llcrypt unregistered [ 1816.008121] Key type ._llcrypt registered [ 1816.010529] Key type .llcrypt registered [ 1816.085229] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1816.798062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1816.814101] alg: No test for adler32 (adler32-zlib) [ 1817.750675] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 1817.889221] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 1819.528725] Key type lgssc registered [ 1820.140965] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1824.752331] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1824.767959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1840.288168] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1845.407453] LustreError: 76107:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.141@tcp: failed processing log, type 4: rc = -110 [ 1872.031187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1875.624835] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1878.889802] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1883.419583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1883.495976] Lustre: *** cfs_fail_loc=706, val=0*** [ 1883.713515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1883.750790] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1886.066598] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1891.318634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1891.528321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1894.175836] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1897.555722] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1899.528494] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1902.051586] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1902.051737] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1902.057870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1902.063244] Lustre: Skipped 1 previous similar message [ 1907.170076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1907.176177] Lustre: Skipped 1 previous similar message [ 1912.289679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1912.293698] Lustre: Skipped 1 previous similar message [ 1915.359179] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1915.461344] Lustre: server umount lustre-OST0000 complete [ 1922.528743] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1922.530906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1922.535827] Lustre: Skipped 1 previous similar message [ 1922.541534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1922.547591] Lustre: Skipped 1 previous similar message [ 1923.755600] Lustre: server umount lustre-MDT0000 complete [ 1925.693599] LustreError: 76113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772437585 with bad export cookie 2199601743579688204 [ 1925.697921] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1925.704762] LustreError: 76113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1925.849285] Lustre: server umount lustre-MDT0001 complete [ 1930.817439] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1932.343578] Key type lgssc unregistered [ 1932.481535] LNet: 79365:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1932.484964] LNetError: 79365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1932.492340] LNet: Removed LNI 192.168.201.141@tcp [ 1932.891774] Key type .llcrypt unregistered [ 1932.894068] Key type ._llcrypt unregistered [ 1945.446119] Key type ._llcrypt registered [ 1945.447612] Key type .llcrypt registered [ 1945.508136] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1946.010984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1946.080434] alg: No test for adler32 (adler32-zlib) [ 1946.994414] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 1947.123621] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 1948.727233] Key type lgssc registered [ 1949.300848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1953.446968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1953.456504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1956.343334] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1960.100660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1960.390924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1962.152500] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1965.537691] LustreError: 81412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1965.545015] LustreError: 81412:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1966.334744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1966.527753] LustreError: 81410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1966.534337] LustreError: 81410:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1966.537381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1969.164307] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1970.656612] LustreError: 81416:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1970.663708] LustreError: 81416:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1972.785735] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1972.790980] Lustre: MGS: Client 4ab0dfa8-f048-4ad3-9434-b58172159376 (at 0@lo) reconnecting [ 1972.802394] LustreError: 81426:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1972.899936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1972.942318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1974.481587] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1983.461096] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1983.462190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1983.464086] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1983.472517] Lustre: Skipped 1 previous similar message [ 1988.580988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1988.583927] Lustre: Skipped 1 previous similar message [ 1989.202937] Lustre: server umount lustre-OST0000 complete [ 1993.698810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1993.699475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1993.701806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1993.701815] Lustre: Skipped 1 previous similar message [ 1993.708566] Lustre: Skipped 1 previous similar message [ 1997.019391] Lustre: server umount lustre-MDT0000 complete [ 1998.877284] Lustre: server umount lustre-MDT0001 complete [ 2000.558313] Lustre: server umount MGS complete [ 2006.645855] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 02:47:45 (1772437665) [ 2018.683368] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 2022.743842] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2037.705280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2039.465444] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2044.039937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2044.141957] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2044.166610] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2044.206541] Lustre: lustre-MDT0001: new disk, initializing [ 2044.242962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2044.253839] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2044.256935] Lustre: 88469:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2044.259695] LustreError: 88469:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2045.879487] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2051.177073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2051.296900] Lustre: lustre-OST0000: new disk, initializing [ 2051.299379] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2051.335748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2054.122345] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2058.078871] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2059.297141] Lustre: 89815:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000b03b2479 x1858535536232704/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1772437735 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2061.516583] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2061.524180] Lustre: MGS: Client a5ee2acf-551d-4579-9915-b8ea3040b4c6 (at 0@lo) reconnecting [ 2061.535863] LustreError: 88454:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2061.551744] Lustre: 87755:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2061.577654] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2061.626854] Lustre: lustre-MDT0000: new disk, initializing [ 2061.698078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2061.710457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2062.751837] Lustre: 90175:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000032c3288 x1858535536238080/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1772437738 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2062.816956] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2063.509401] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2064.809952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2064.817338] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2064.886565] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2083.808118] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2083.811419] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2083.817300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2083.819440] Lustre: Skipped 1 previous similar message [ 2089.381965] Lustre: server umount lustre-OST0000 complete [ 2095.585478] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2095.586095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2095.589744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2095.589751] Lustre: Skipped 2 previous similar messages [ 2095.591089] Lustre: Skipped 2 previous similar messages [ 2097.358367] Lustre: server umount lustre-MDT0000 complete [ 2105.150807] Lustre: server umount lustre-MDT0001 complete [ 2107.031163] Lustre: server umount MGS complete [ 2112.659987] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 02:49:31 (1772437771) [ 2121.355721] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 2122.856735] Key type lgssc unregistered [ 2123.029611] LNet: 92635:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2123.033627] LNetError: 92635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2123.049658] LNet: Removed LNI 192.168.201.141@tcp [ 2123.529813] Key type .llcrypt unregistered [ 2123.531482] Key type ._llcrypt unregistered [ 2132.649894] Key type ._llcrypt registered [ 2132.651363] Key type .llcrypt registered [ 2132.697993] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2133.270425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2133.297880] alg: No test for adler32 (adler32-zlib) [ 2134.187506] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 2134.306197] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 2135.919150] Key type lgssc registered [ 2136.508070] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2140.509305] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2140.523092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2141.752420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2143.698165] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2148.276445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2148.473437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2150.298065] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2153.089156] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2154.858534] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2158.116182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2158.261497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2160.438966] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2162.479497] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2166.437484] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2171.873427] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2171.874702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2171.878790] Lustre: Skipped 1 previous similar message [ 2171.882810] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2176.992441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2176.996064] Lustre: Skipped 2 previous similar messages [ 2177.743472] Lustre: server umount lustre-OST0000 complete [ 2179.551900] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2179.554132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2179.558705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2182.112745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2182.112984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2187.232505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2187.234896] Lustre: Skipped 2 previous similar messages [ 2193.376357] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2193.470694] Lustre: server umount lustre-MDT0000 complete [ 2195.248396] LustreError: 95974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772437855 with bad export cookie 13695848461689212687 [ 2195.252123] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2195.253124] LustreError: 95974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2195.380974] Lustre: server umount lustre-MDT0001 complete [ 2199.537098] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 2200.770528] Key type lgssc unregistered [ 2200.927656] LNet: 96996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2200.931193] LNetError: 96996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2200.940224] LNet: Removed LNI 192.168.201.141@tcp [ 2201.302900] Key type .llcrypt unregistered [ 2201.304314] Key type ._llcrypt unregistered [ 2208.741580] Key type ._llcrypt registered [ 2208.744478] Key type .llcrypt registered [ 2208.795141] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2209.340782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2209.353187] alg: No test for adler32 (adler32-zlib) [ 2210.241285] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 2210.348763] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 2211.960285] Key type lgssc registered [ 2212.513724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2215.228297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2215.230301] LustreError: 98040:0:(tgt_mount.c:2535:server_fill_super()) cfs_race id 716 sleeping [ 2215.231676] Lustre: Skipped 1 previous similar message [ 2215.239189] LustreError: 98039:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 waking [ 2215.241665] LustreError: 98040:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 2215.244485] LustreError: 98040:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2215.249217] LustreError: 98040:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2215.257498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2216.558219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2220.373648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2220.567930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2222.542268] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2224.814365] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing lsmod [ 2226.202429] LustreError: 99211:0:(tgt_mount.c:2535:server_fill_super()) cfs_race id 716 sleeping [ 2226.208729] LustreError: 99212:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 waking [ 2226.214804] LustreError: 99211:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 awake: rc=4993 [ 2226.222116] LustreError: 99211:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2226.228491] LustreError: 99211:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2226.233497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2226.485716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2227.502856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2230.752852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2230.757334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2230.767095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2231.778843] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2231.780263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2231.784727] Lustre: Skipped 1 previous similar message [ 2231.787322] Lustre: Skipped 1 previous similar message [ 2233.972442] Lustre: server umount lustre-MDT0000 complete [ 2235.541290] LustreError: 99250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772437895 with bad export cookie 17071695671263985937 [ 2235.544095] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2235.554384] LustreError: 99250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2235.831332] LustreError: 98508:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000a6c4540a x1858535811991040/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2235.907584] Lustre: server umount lustre-MDT0001 complete [ 2247.779801] Lustre: server umount lustre-OST0000 complete [ 2251.280385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2251.466367] LustreError: 100278:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2251.475516] LustreError: 100278:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2251.496906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2253.132262] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2256.744059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2256.863773] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2256.867151] Lustre: Skipped 1 previous similar message [ 2256.921262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2258.669943] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2261.190945] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2263.128119] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2266.316794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2266.506631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2268.880200] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2271.199782] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2276.774261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 2287.075032] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2287.077520] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2287.079532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2287.084140] Lustre: Skipped 1 previous similar message [ 2290.534417] Lustre: server umount lustre-OST0000 complete [ 2292.193190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2292.194665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2292.198940] Lustre: Skipped 1 previous similar message [ 2292.206161] Lustre: Skipped 2 previous similar messages [ 2298.217594] Lustre: server umount lustre-MDT0000 complete [ 2299.939426] LustreError: 100259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772437960 with bad export cookie 17071695671263986854 [ 2299.942519] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2299.943275] LustreError: 100259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2300.076145] Lustre: server umount lustre-MDT0001 complete [ 2304.305697] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 2305.507106] Key type lgssc unregistered [ 2305.659398] LNet: 103480:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2305.662354] LNetError: 103480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2305.670381] LNet: Removed LNI 192.168.201.141@tcp [ 2306.008855] Key type .llcrypt unregistered [ 2306.010481] Key type ._llcrypt unregistered [ 2315.774594] Key type ._llcrypt registered [ 2315.776473] Key type .llcrypt registered [ 2315.822197] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2316.154236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2316.179856] alg: No test for adler32 (adler32-zlib) [ 2317.033910] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 2317.123447] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 2318.711137] Key type lgssc registered [ 2319.106825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2322.257328] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2322.270763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2323.547410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2325.186959] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2328.952613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2329.140366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2330.733050] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2333.002153] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2334.417486] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2336.995528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2337.133790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2339.314608] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2341.381136] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2348.391111] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 2355.273837] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2356.726363] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2358.753402] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2358.755517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2358.757560] Lustre: Skipped 1 previous similar message [ 2363.880341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2363.883570] Lustre: Skipped 2 previous similar messages [ 2364.180318] Lustre: server umount lustre-OST0000 complete [ 2368.994639] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2368.995320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2368.999747] Lustre: Skipped 1 previous similar message [ 2371.890456] Lustre: server umount lustre-MDT0000 complete [ 2373.514199] LustreError: 104837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772438033 with bad export cookie 1778179025396677807 [ 2373.517851] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2373.518033] LustreError: 104837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2373.650457] Lustre: server umount lustre-MDT0001 complete [ 2377.799415] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 2378.953863] Key type lgssc unregistered [ 2379.095640] LNet: 108157:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2379.102136] LNetError: 108157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2379.111686] LNet: Removed LNI 192.168.201.141@tcp [ 2379.472967] Key type .llcrypt unregistered [ 2379.474128] Key type ._llcrypt unregistered [ 2386.869939] Key type ._llcrypt registered [ 2386.871643] Key type .llcrypt registered [ 2386.912560] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2387.359421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2387.365937] alg: No test for adler32 (adler32-zlib) [ 2388.232409] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 2388.336215] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 2389.919143] Key type lgssc registered [ 2390.359852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2393.540603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2393.552506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2394.790746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2396.478704] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2400.268769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2400.426833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2402.043477] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2404.524807] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2406.065193] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2408.873037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2409.043780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2411.420309] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2413.804525] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2419.308400] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 2424.802570] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2424.803518] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2424.807127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2424.811632] Lustre: Skipped 1 previous similar message [ 2428.352869] Lustre: server umount lustre-OST0000 complete [ 2431.456093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2431.459152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2431.463895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2431.465940] Lustre: Skipped 1 previous similar message [ 2435.042819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2435.043624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2436.320816] Lustre: server umount lustre-MDT0000 complete [ 2437.880440] LustreError: 109321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772438097 with bad export cookie 12576497680271308791 [ 2437.884530] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2437.885828] LustreError: 109321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2438.016729] Lustre: server umount lustre-MDT0001 complete [ 2442.493950] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 2443.984940] Key type lgssc unregistered [ 2444.133320] LNet: 112518:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2444.136100] LNetError: 112518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2444.145448] LNet: Removed LNI 192.168.201.141@tcp [ 2444.536253] Key type .llcrypt unregistered [ 2444.537727] Key type ._llcrypt unregistered [ 2456.122813] Key type ._llcrypt registered [ 2456.124624] Key type .llcrypt registered [ 2456.179829] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2456.744824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2456.757777] alg: No test for adler32 (adler32-zlib) [ 2457.637605] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 2457.744474] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 2459.343962] Key type lgssc registered [ 2459.823690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2463.154372] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2463.172578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2464.440331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2466.316319] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2470.009215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2470.185217] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2471.943854] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2474.254383] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2475.941792] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2479.052139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2479.179243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2481.329595] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2483.465541] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2489.447626] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 2497.186310] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2498.764369] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2498.766728] Lustre: Skipped 2 previous similar messages [ 2500.542306] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2500.545605] Lustre: Skipped 2 previous similar messages [ 2504.185381] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2504.195347] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2507.349902] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.41@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2507.353873] Lustre: Skipped 2 previous similar messages [ 2511.418814] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.41@tcp 192.168.0.[2,10]@tcp [ 2515.312781] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.41/24@tcp in log lustre-MDT0000 [ 2515.316940] Lustre: Skipped 2 previous similar messages [ 2521.084657] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.41/24@tcp [ 2521.087419] Lustre: Skipped 1 previous similar message [ 2525.668754] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2525.669412] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2525.670137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2525.680075] Lustre: Skipped 1 previous similar message [ 2528.729226] Lustre: server umount lustre-OST0000 complete [ 2530.785129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2530.785787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2530.790598] Lustre: Skipped 1 previous similar message [ 2530.792871] Lustre: Skipped 1 previous similar message [ 2535.906894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2535.910510] Lustre: Skipped 2 previous similar messages [ 2536.478845] Lustre: server umount lustre-MDT0000 complete [ 2538.053292] LustreError: 116210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772438198 with bad export cookie 17235328510421790379 [ 2538.055719] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2538.059391] LustreError: 116210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2538.201756] Lustre: server umount lustre-MDT0001 complete [ 2542.607086] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 2543.759878] Key type lgssc unregistered [ 2543.894823] LNet: 118591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2543.898247] LNetError: 118591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2543.908476] LNet: Removed LNI 192.168.201.141@tcp [ 2544.240477] Key type .llcrypt unregistered [ 2544.241985] Key type ._llcrypt unregistered [ 2555.524636] Key type ._llcrypt registered [ 2555.525872] Key type .llcrypt registered [ 2555.567605] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2555.935923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2555.985210] alg: No test for adler32 (adler32-zlib) [ 2556.868972] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 2556.977063] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 2558.575239] Key type lgssc registered [ 2559.066810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2562.042096] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2562.055193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2563.293543] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2563.298895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2563.304078] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.41/24@tcp [ 2564.812580] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2568.102155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2568.232766] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2568.243280] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.41/24@tcp [ 2568.267528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2569.751042] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2571.829748] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2573.185816] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2575.694824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2575.845087] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2578.175423] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2580.350508] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2589.160727] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 2599.392592] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2599.394026] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2599.396119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2599.396843] LustreError: Skipped 1 previous similar message [ 2599.406646] Lustre: Skipped 1 previous similar message [ 2600.408673] Lustre: server umount lustre-OST0000 complete [ 2603.999958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2604.003575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2604.008850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2604.010740] Lustre: Skipped 1 previous similar message [ 2609.632177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2609.634712] Lustre: Skipped 3 previous similar messages [ 2614.752529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2614.755245] Lustre: Skipped 1 previous similar message [ 2616.287113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2616.348209] Lustre: server umount lustre-MDT0000 complete [ 2617.745319] LustreError: 120092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772438277 with bad export cookie 14769513313600577742 [ 2617.746988] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2617.752309] LustreError: 120092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2617.873260] Lustre: server umount lustre-MDT0001 complete [ 2621.739702] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 2622.804545] Key type lgssc unregistered [ 2622.928342] LNet: 123365:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2622.931096] LNetError: 123365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2622.940281] LNet: Removed LNI 192.168.201.141@tcp [ 2623.247711] Key type .llcrypt unregistered [ 2623.248589] Key type ._llcrypt unregistered [ 2632.629610] Key type ._llcrypt registered [ 2632.631327] Key type .llcrypt registered [ 2632.681737] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 2639.320821] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2639.755383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2639.765698] alg: No test for adler32 (adler32-zlib) [ 2640.615435] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 2640.700499] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 2642.287164] Key type lgssc registered [ 2642.715121] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2656.181447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2656.189180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2657.289714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2657.304691] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2657.349307] Lustre: lustre-MDT0000: new disk, initializing [ 2657.379206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2657.386683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2659.161588] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2663.638062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2663.683151] Lustre: 126703:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2663.708316] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2663.710677] Lustre: Skipped 1 previous similar message [ 2663.751350] Lustre: lustre-MDT0001: new disk, initializing [ 2663.787527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2663.801548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2663.806301] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2665.195156] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2668.703873] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2670.157233] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2672.793840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2672.911438] Lustre: lustre-OST0000: new disk, initializing [ 2672.913742] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2672.949683] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2673.127189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2673.131951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2673.180023] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2675.116284] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2678.447925] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2679.955470] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2680.039035] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2681.630282] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2681.719426] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2683.360382] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2683.362404] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2683.363095] LustreError: Skipped 1 previous similar message [ 2683.363692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2683.367289] Lustre: Skipped 1 previous similar message [ 2688.484735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2688.488461] Lustre: Skipped 2 previous similar messages [ 2688.556318] Lustre: server umount lustre-OST0000 complete [ 2693.600604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2693.601050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2693.605907] Lustre: Skipped 1 previous similar message [ 2698.719988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2698.721940] Lustre: Skipped 2 previous similar messages [ 2703.839863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2703.841780] Lustre: Skipped 1 previous similar message [ 2704.351281] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2704.420701] Lustre: server umount lustre-MDT0000 complete [ 2705.834322] LustreError: 126694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772438365 with bad export cookie 15677235292342492910 [ 2705.837462] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2705.840666] LustreError: 126694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2705.971200] Lustre: server umount lustre-MDT0001 complete [ 2709.374844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2709.601558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2710.997820] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2714.279729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2714.403294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2715.870748] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2717.802690] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2719.117031] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2721.694166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2723.829631] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2725.887965] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2727.363607] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2734.577884] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 2735.985979] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2736.066238] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2739.622212] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2739.682695] Lustre: lustre-OST0001: new disk, initializing [ 2739.684602] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2739.715460] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2739.718707] Lustre: Skipped 1 previous similar message [ 2741.580162] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2745.226188] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2747.901920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2747.905845] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2747.936114] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2748.379758] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2749.868228] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2749.946741] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2751.385761] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff959b03747800.ost_server_uuid 50 [ 2751.981953] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff959b03747800.ost_server_uuid in FULL state after 0 sec [ 2758.118679] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2758.118764] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2758.121886] LustreError: Skipped 1 previous similar message [ 2758.127676] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2758.131422] Lustre: Skipped 1 previous similar message [ 2760.448098] Lustre: server umount lustre-OST0001 complete [ 2763.235959] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2763.236880] LustreError: 132569:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2763.240560] Lustre: Skipped 2 previous similar messages [ 2763.246176] LustreError: 132569:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2768.355132] LustreError: 132099:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2768.360362] LustreError: 132099:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2773.473944] LustreError: 132569:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2773.481191] LustreError: 132569:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2776.031117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2776.085753] Lustre: server umount lustre-OST0000 complete [ 2778.594126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2778.595890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2778.598763] Lustre: Skipped 1 previous similar message [ 2778.600320] Lustre: Skipped 6 previous similar messages [ 2783.584370] Lustre: server umount lustre-MDT0000 complete [ 2783.711971] LustreError: 131219:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2783.716689] LustreError: 131219:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2784.798626] LustreError: 133557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772438444 with bad export cookie 15677235292342493778 [ 2784.801282] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2784.804263] LustreError: 133557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2784.915445] Lustre: server umount lustre-MDT0001 complete [ 2791.441731] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 2792.465333] Key type lgssc unregistered [ 2792.605449] LNet: 136414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2792.607865] LNetError: 136414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2792.616322] LNet: Removed LNI 192.168.201.141@tcp [ 2792.915680] Key type .llcrypt unregistered [ 2792.917327] Key type ._llcrypt unregistered [ 2806.701431] Key type ._llcrypt registered [ 2806.702638] Key type .llcrypt registered [ 2806.738938] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 2813.205066] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2813.579622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2813.609193] alg: No test for adler32 (adler32-zlib) [ 2814.462148] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 2814.551046] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 2816.135162] Key type lgssc registered [ 2816.583996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2829.041202] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2829.047879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2830.151510] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2830.163850] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2830.202650] Lustre: lustre-MDT0000: new disk, initializing [ 2830.231154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2830.239688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2831.714576] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2836.465298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2836.517330] Lustre: 140519:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2836.542667] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2836.545478] Lustre: Skipped 1 previous similar message [ 2836.587021] Lustre: lustre-MDT0001: new disk, initializing [ 2836.625606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2836.641819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2836.645380] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2838.159732] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2841.928068] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2843.877489] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2846.946906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2847.078488] Lustre: lustre-OST0000: new disk, initializing [ 2847.080960] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2847.117895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2849.527269] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2852.845450] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2854.403387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2854.406951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2854.447355] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2860.246556] Lustre: Failing over lustre-OST0000 [ 2860.307466] Lustre: server umount lustre-OST0000 complete [ 2873.463228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2873.572341] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2873.582645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2873.589915] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2874.667472] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2875.600897] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2876.944297] Lustre: Failing over lustre-MDT0000 [ 2877.046771] Lustre: server umount lustre-MDT0000 complete [ 2877.919757] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2877.922205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2877.925634] Lustre: Skipped 1 previous similar message [ 2878.945067] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2878.949245] Lustre: Skipped 1 previous similar message [ 2879.967128] Lustre: 138749:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772438524/real 1772438524] req@000000009d02f4e2 x1858536445342464/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1772438540 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2879.974841] Lustre: 138749:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2884.064173] LustreError: 140532:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2884.072510] LustreError: 140532:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2884.894532] LustreError: 140526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2885.087189] Lustre: 138747:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772438529/real 1772438529] req@00000000e35dc1d9 x1858536445343360/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1772438545 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2889.184035] LustreError: 141305:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2889.190284] LustreError: 141305:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2890.196300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2890.244516] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2890.249096] Lustre: 140523:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.201.141@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2890.331162] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2890.357203] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2890.357738] Lustre: lustre-OST0000: Recovery over after 0:16, of 3 clients 3 recovered and 0 were evicted. [ 2890.361519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2890.399474] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2891.785857] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2895.135315] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2895.849052] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2895.851065] Lustre: Skipped 1 previous similar message [ 2895.854837] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2895.872294] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2900.963074] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2900.963186] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2900.963187] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2900.963722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2900.966949] Lustre: Skipped 1 previous similar message [ 2903.055812] Lustre: server umount lustre-OST0000 complete [ 2906.080557] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2906.080950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2906.081519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2906.081522] Lustre: Skipped 1 previous similar message [ 2906.091938] Lustre: Skipped 1 previous similar message [ 2910.550371] Lustre: server umount lustre-MDT0000 complete [ 2911.200064] LustreError: 141305:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2911.207616] LustreError: 141305:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2911.842245] LustreError: 140511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772438571 with bad export cookie 8780719249759337719 [ 2911.845620] LustreError: 140511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2911.846182] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2911.957304] Lustre: server umount lustre-MDT0001 complete [ 2915.584921] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 2916.644249] Key type lgssc unregistered [ 2916.792362] LNet: 145749:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2916.795615] LNetError: 145749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2916.803452] LNet: Removed LNI 192.168.201.141@tcp [ 2917.103237] Key type .llcrypt unregistered [ 2917.104388] Key type ._llcrypt unregistered [ 2937.581680] Key type ._llcrypt registered [ 2937.582904] Key type .llcrypt registered [ 2937.621027] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2938.105474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2938.115532] alg: No test for adler32 (adler32-zlib) [ 2938.977571] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 2939.092591] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 2940.679147] Key type lgssc registered [ 2941.083038] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2944.027120] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2944.034494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2945.219017] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2946.558733] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2949.803836] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2951.937583] LustreError: 147279:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2951.941516] LustreError: 147279:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2951.944378] LustreError: 147279:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 2952.073734] Lustre: server umount lustre-MDT0000 complete [ 2957.207886] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 2960.625695] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2973.438596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2973.530822] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2973.553340] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2973.588540] Lustre: lustre-MDT0000: new disk, initializing [ 2973.612434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2973.620060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2974.952967] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2979.209227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2979.250546] Lustre: 150430:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2979.282190] LustreError: 150439:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2979.286643] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2979.288454] LustreError: 150439:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2979.290828] Lustre: Skipped 1 previous similar message [ 2979.330340] Lustre: lustre-MDT0001: new disk, initializing [ 2979.366617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2979.380169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2979.383699] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2980.807063] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2984.103802] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2985.450988] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2987.856798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2987.964660] Lustre: lustre-OST0000: new disk, initializing [ 2987.967196] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2988.000575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2989.961362] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2990.282683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2990.286219] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2990.328594] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2993.149795] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2995.069529] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2995.702708] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 03:04:15 (1772438655) [ 3071.455568] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3071.458677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3071.463574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3072.479372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3072.479580] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3072.482307] Lustre: Skipped 1 previous similar message [ 3072.487514] Lustre: Skipped 1 previous similar message [ 3075.808183] Lustre: server umount lustre-MDT0000 complete [ 3077.045560] LustreError: 150420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772438737 with bad export cookie 15779886059279675886 [ 3077.047052] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3077.049950] LustreError: 150420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3077.599346] LustreError: 150434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3077.599419] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3077.607501] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3077.609519] Lustre: Skipped 1 previous similar message [ 3082.719531] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3082.719531] LustreError: 151721:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3083.236331] Lustre: server umount lustre-MDT0001 complete [ 3090.638072] Lustre: server umount lustre-OST0000 complete [ 3093.425389] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 3096.345289] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 3107.702327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3107.803283] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3107.816286] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3107.854939] Lustre: lustre-MDT0000: new disk, initializing [ 3107.879504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3107.887442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3109.122706] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3113.143777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3113.182596] Lustre: 156648:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3113.204177] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3113.205748] Lustre: Skipped 1 previous similar message [ 3113.244368] Lustre: lustre-MDT0001: new disk, initializing [ 3113.305892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3113.310277] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3114.471958] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3117.440725] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3118.582220] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3120.691428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3120.824703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3120.826527] Lustre: Skipped 1 previous similar message [ 3122.566155] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3125.447462] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3125.748264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3125.751934] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3125.784295] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3126.644794] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3126.705211] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3127.905760] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3127.975785] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3130.849129] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3130.850997] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3130.851619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3130.855054] Lustre: Skipped 1 previous similar message [ 3130.858185] Lustre: Skipped 1 previous similar message [ 3134.674222] Lustre: server umount lustre-OST0000 complete [ 3135.968032] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3135.971116] Lustre: Skipped 1 previous similar message [ 3141.088042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3141.089464] Lustre: Skipped 3 previous similar messages [ 3150.303121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3150.359979] Lustre: server umount lustre-MDT0000 complete [ 3151.328126] LustreError: 156660:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3151.331782] LustreError: 156660:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3151.491074] LustreError: 157425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772438811 with bad export cookie 15779886059279834716 [ 3151.494202] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3151.495476] LustreError: 157425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3151.602298] Lustre: server umount lustre-MDT0001 complete [ 3154.182147] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 03:06:53 (1772438813) [ 3159.459652] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 3162.410405] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 3173.842238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3173.933644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3173.946274] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3173.949250] Lustre: Skipped 1 previous similar message [ 3173.983334] Lustre: lustre-MDT0000: new disk, initializing [ 3173.985272] Lustre: Skipped 1 previous similar message [ 3174.008527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3174.014577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3175.249723] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3179.406647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3179.451210] Lustre: 162960:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3179.558427] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3180.844558] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3183.934706] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3185.228497] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3187.539194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3189.443381] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3192.468635] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3194.361192] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3194.402719] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3204.579996] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3204.580936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3204.585020] Lustre: Skipped 1 previous similar message [ 3204.587388] Lustre: Skipped 2 previous similar messages [ 3204.590741] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3204.593101] LustreError: Skipped 1 previous similar message [ 3205.643777] Lustre: server umount lustre-OST0000 complete [ 3213.085918] Lustre: server umount lustre-MDT0000 complete [ 3214.381230] LustreError: 163742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772438874 with bad export cookie 15779886059279835584 [ 3214.385061] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3214.385168] LustreError: 163742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3214.508373] Lustre: server umount lustre-MDT0001 complete [ 3217.482450] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 03:07:56 (1772438876) [ 3222.941241] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 3226.063021] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 3238.417920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3238.508964] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3238.522212] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3238.524094] Lustre: Skipped 3 previous similar messages [ 3238.558730] Lustre: lustre-MDT0000: new disk, initializing [ 3238.560094] Lustre: Skipped 2 previous similar messages [ 3238.587808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3238.590408] Lustre: Skipped 2 previous similar messages [ 3238.596554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3238.599019] Lustre: Skipped 2 previous similar messages [ 3239.853490] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3244.105907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3244.153692] Lustre: 168875:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3244.263378] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3245.512982] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3248.554841] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3249.762611] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3251.909635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3253.880871] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3256.896324] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3258.897778] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3263.968827] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3263.970138] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3263.974433] Lustre: Skipped 2 previous similar messages [ 3263.976069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3263.977635] Lustre: Skipped 4 previous similar messages [ 3278.303103] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3278.360847] Lustre: server umount lustre-OST0000 complete [ 3280.351743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3280.353990] LustreError: Skipped 1 previous similar message [ 3285.721513] Lustre: server umount lustre-MDT0000 complete [ 3287.007125] LustreError: 168867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772438947 with bad export cookie 15779886059279836641 [ 3287.011028] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3287.011890] LustreError: 168867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3287.127137] Lustre: server umount lustre-MDT0001 complete [ 3290.560354] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 3291.583825] Key type lgssc unregistered [ 3291.713408] LNet: 172349:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3291.716326] LNetError: 172349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3291.725605] LNet: Removed LNI 192.168.201.141@tcp [ 3292.017361] Key type .llcrypt unregistered [ 3292.018658] Key type ._llcrypt unregistered [ 3301.007129] Key type ._llcrypt registered [ 3301.008204] Key type .llcrypt registered [ 3301.046869] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 3301.390595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3301.423318] alg: No test for adler32 (adler32-zlib) [ 3302.300621] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 3302.400761] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 3303.983151] Key type lgssc registered [ 3304.355235] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3307.253090] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3307.260615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3308.444930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3309.827688] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3312.863448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3313.009515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3314.449916] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3316.289209] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3317.460925] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3319.720092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3319.834262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3321.642399] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3323.353409] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3333.158748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3348.451564] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3348.452629] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3348.452942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3348.456336] Lustre: Skipped 1 previous similar message [ 3351.643250] Lustre: server umount lustre-OST0000 complete [ 3353.570751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3353.573767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3353.575040] Lustre: Skipped 1 previous similar message [ 3353.578035] Lustre: Skipped 2 previous similar messages [ 3358.687817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3358.689572] Lustre: Skipped 1 previous similar message [ 3363.808070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3363.810195] Lustre: Skipped 1 previous similar message [ 3367.391170] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3367.448848] Lustre: server umount lustre-MDT0000 complete [ 3368.718479] LustreError: 175339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772439028 with bad export cookie 13973529843634418274 [ 3368.722270] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3368.722402] LustreError: 175339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3368.842735] Lustre: server umount lustre-MDT0001 complete [ 3372.477585] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 3373.580062] Key type lgssc unregistered [ 3373.721422] LNet: 176927:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3373.724623] LNetError: 176927:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3373.735349] LNet: Removed LNI 192.168.201.141@tcp [ 3374.056311] Key type .llcrypt unregistered [ 3374.057508] Key type ._llcrypt unregistered [ 3383.021655] Key type ._llcrypt registered [ 3383.022945] Key type .llcrypt registered [ 3383.061218] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 3383.382617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3383.420031] alg: No test for adler32 (adler32-zlib) [ 3384.293387] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 3384.384390] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 3385.975118] Key type lgssc registered [ 3386.333292] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3388.892574] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3388.902506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3390.061947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3391.440334] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3394.278167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3394.400855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3395.715309] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3397.732800] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3398.954231] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3401.265333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3401.386885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3403.239136] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3405.094395] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3415.715373] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3420.128718] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3420.131827] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3420.137332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3420.966269] Lustre: lustre-OST0000: Not available for connect from 192.168.201.41@tcp (stopping) [ 3421.154643] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3421.159496] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3424.145305] Lustre: server umount lustre-OST0000 complete [ 3425.875954] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff959b0a63c000.ost_server_uuid 50 [ 3426.362474] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff959b0a63c000.ost_server_uuid in DISCONN state after 0 sec [ 3426.892437] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3440.607921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3440.610168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3440.614382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3440.615950] Lustre: Skipped 2 previous similar messages [ 3443.094280] Lustre: server umount lustre-MDT0000 complete [ 3444.350329] LustreError: 178284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772439104 with bad export cookie 9994661108443283226 [ 3444.353159] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3444.353749] LustreError: 178284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3444.463183] Lustre: server umount lustre-MDT0001 complete [ 3447.357315] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 03:11:46 (1772439106) [ 3449.493179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3449.661354] LustreError: 181833:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3449.668863] LustreError: 181833:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3449.686949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3450.991674] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3453.721670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3453.842099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3455.055951] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3456.876648] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3458.078113] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3460.286368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3462.337169] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3464.119278] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3466.365172] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3466.443644] Lustre: lustre-OST0001: new disk, initializing [ 3466.445861] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3466.479160] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3466.480957] Lustre: Skipped 1 previous similar message [ 3468.396075] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3470.846196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3470.849177] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3470.875628] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3471.463535] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3473.887689] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3473.890132] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3473.893878] Lustre: Skipped 1 previous similar message [ 3473.895623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3473.897722] Lustre: Skipped 2 previous similar messages [ 3478.390656] Lustre: server umount lustre-OST0000 complete [ 3480.035850] LustreError: 183427:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3480.042545] LustreError: 183427:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3480.287211] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3480.368269] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3481.714335] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3481.792283] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3481.888349] LustreError: 183425:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3485.152245] LustreError: 183427:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3485.158464] LustreError: 183427:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3490.272903] LustreError: 183428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3490.278432] LustreError: 183428:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3495.394042] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3495.395906] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3495.398466] Lustre: Skipped 1 previous similar message [ 3495.401399] Lustre: Skipped 4 previous similar messages [ 3497.874696] Lustre: server umount lustre-OST0001 complete [ 3499.999687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3505.631995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3505.634670] Lustre: Skipped 4 previous similar messages [ 3513.311091] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3513.371386] Lustre: server umount lustre-MDT0000 complete [ 3514.569190] LustreError: 182552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772439174 with bad export cookie 9994661108443284290 [ 3514.572955] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3514.573104] LustreError: 182552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3514.698595] Lustre: server umount lustre-MDT0001 complete [ 3522.178305] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 03:13:01 (1772439181) [ 3524.322991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3524.406718] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3524.422981] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3524.489170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3525.735244] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3528.619324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3528.655538] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3528.674160] Lustre: 187606:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3530.124568] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3531.971064] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3533.220190] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3535.567679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3535.633446] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3537.603148] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3539.222429] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3539.523288] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3541.788865] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3541.818180] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3543.743217] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3545.533362] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3545.746379] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 3549.663967] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3549.664127] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3549.665955] LustreError: Skipped 1 previous similar message [ 3549.669250] Lustre: Skipped 2 previous similar messages [ 3549.671649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3549.674413] Lustre: Skipped 3 previous similar messages [ 3552.445343] Lustre: server umount lustre-OST0000 complete [ 3556.129210] LustreError: 189224:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3556.134543] LustreError: 189224:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3566.562659] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3566.563018] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3566.567036] Lustre: Skipped 1 previous similar message [ 3566.569681] LustreError: Skipped 1 previous similar message [ 3569.300505] Lustre: server umount lustre-OST0001 complete [ 3576.733127] Lustre: server umount lustre-MDT0000 complete [ 3576.801051] LustreError: 187616:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3576.806099] LustreError: 187616:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 3577.958291] LustreError: 187597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772439238 with bad export cookie 9994661108443285452 [ 3577.961162] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3577.968502] LustreError: 187597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3578.071880] Lustre: server umount lustre-MDT0001 complete [ 3586.169634] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 03:14:05 (1772439245) [ 3591.844868] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 3595.194807] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 3606.926599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3607.020134] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3607.032836] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3607.068930] Lustre: lustre-MDT0000: new disk, initializing [ 3607.092712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3607.094772] Lustre: Skipped 3 previous similar messages [ 3607.099487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3608.349437] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3612.400560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3612.443266] Lustre: 195174:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3612.465816] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3612.467252] Lustre: Skipped 1 previous similar message [ 3612.502275] Lustre: lustre-MDT0001: new disk, initializing [ 3612.546157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3612.550278] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3613.807072] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3616.905667] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3618.053908] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3620.195633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3620.295317] Lustre: lustre-OST0000: new disk, initializing [ 3620.296982] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3622.100429] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3624.310521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3624.313490] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3624.345879] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3624.979855] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3626.186438] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3626.252373] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3627.447221] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3627.519898] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3629.536404] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3629.537469] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3629.539680] LustreError: Skipped 1 previous similar message [ 3629.543591] Lustre: Skipped 2 previous similar messages [ 3629.546203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3629.548768] Lustre: Skipped 8 previous similar messages [ 3630.226236] Lustre: server umount lustre-OST0000 complete [ 3637.658347] Lustre: server umount lustre-MDT0000 complete [ 3638.963362] LustreError: 195166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772439299 with bad export cookie 9994661108443287216 [ 3638.966437] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3638.968612] LustreError: 195166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3639.095859] Lustre: server umount lustre-MDT0001 complete [ 3642.072092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3642.233275] LustreError: 198980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3643.581438] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3646.403906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3647.892575] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3649.759286] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3650.979944] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3653.290836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3655.333057] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3657.143292] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3658.537216] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3668.803241] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 3670.068738] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3670.130809] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3676.877808] Lustre: server umount lustre-OST0000 complete [ 3678.748046] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3680.061547] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3703.708528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3703.837360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3703.840487] Lustre: Skipped 5 previous similar messages [ 3705.739982] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3707.524591] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3708.064192] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3711.052134] Lustre: server umount lustre-OST0000 complete [ 3712.239366] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3712.240024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3712.243554] Lustre: Skipped 5 previous similar messages [ 3712.245414] Lustre: Skipped 6 previous similar messages [ 3714.321864] Lustre: server umount lustre-MDT0000 complete [ 3715.483180] LustreError: 198960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772439375 with bad export cookie 9994661108443288084 [ 3715.486163] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3715.486301] LustreError: 198960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3715.597876] Lustre: server umount lustre-MDT0001 complete [ 3718.203379] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 03:16:17 (1772439377) [ 3720.308922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3720.443659] LustreError: 204489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3721.698904] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3724.270170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3725.537139] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3727.202948] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3728.373915] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3730.461045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3732.260370] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3733.868083] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3735.103519] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3745.341378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 3746.526572] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3746.591513] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3748.606583] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3748.670074] Lustre: lustre-OST0001: new disk, initializing [ 3748.671723] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3750.485764] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3753.462742] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3754.694590] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3757.052506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3757.055906] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3757.078079] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3757.804554] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3758.923255] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3758.988048] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3762.149446] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3762.151172] LustreError: Skipped 4 previous similar messages [ 3765.696477] Lustre: server umount lustre-OST0001 complete [ 3767.782749] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3767.875273] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3769.265562] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3769.343978] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3792.955588] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3794.952476] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3796.654385] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3797.153833] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3803.830853] Lustre: server umount lustre-OST0001 complete [ 3830.239103] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3830.290531] Lustre: server umount lustre-OST0000 complete [ 3837.719033] Lustre: server umount lustre-MDT0000 complete [ 3838.922279] LustreError: 204471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772439499 with bad export cookie 9994661108443289001 [ 3838.925421] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3838.926208] LustreError: 204471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3853.279125] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3853.357261] Lustre: server umount lustre-MDT0001 complete [ 3860.840644] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 03:18:40 (1772439520) [ 3863.011473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3863.100345] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3863.109221] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3863.176855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3863.179396] Lustre: Skipped 5 previous similar messages [ 3864.346210] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3867.154308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3867.185716] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3867.196987] Lustre: 212530:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3868.498465] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3870.137197] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3871.251631] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3873.281721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3873.340531] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3875.022805] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3876.635902] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3885.741887] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3885.766710] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3887.583120] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3889.230910] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3890.732620] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 3891.941221] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3892.005746] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3893.204705] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3893.276046] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3894.400713] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff959b1078e000.ost_server_uuid 50 [ 3894.864480] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff959b1078e000.ost_server_uuid in FULL state after 0 sec [ 3895.190988] Lustre: Permanently deactivating lustre-OST0001 [ 3895.196919] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3895.924760] Lustre: Permanently reactivating lustre-OST0001 [ 3901.413171] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3901.416368] LustreError: Skipped 3 previous similar messages [ 3901.418226] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3901.422273] Lustre: Skipped 6 previous similar messages [ 3901.424650] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3901.426976] Lustre: Skipped 19 previous similar messages [ 3902.908412] Lustre: server umount lustre-OST0001 complete [ 3906.531573] LustreError: 214144:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3906.538429] LustreError: 214144:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 28 previous similar messages [ 3910.221272] Lustre: server umount lustre-OST0000 complete [ 3917.605500] Lustre: server umount lustre-MDT0000 complete [ 3918.757940] LustreError: 212565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772439578 with bad export cookie 9994661108443290261 [ 3918.760623] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3918.761562] LustreError: 212565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3918.865528] Lustre: server umount lustre-MDT0001 complete [ 3926.046321] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 03:19:45 (1772439585) [ 3928.280697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3928.369581] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3928.377754] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3928.380571] Lustre: Skipped 5 previous similar messages [ 3929.573735] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3932.157930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3932.192588] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3932.202339] Lustre: 218627:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3933.631920] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3935.377029] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3936.561122] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3938.842776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3938.926282] Lustre: Permanently deactivating lustre-OST0000 [ 3938.931344] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3940.942492] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3942.811942] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3944.434261] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3944.437475] LustreError: 221064:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0001: cannot cleanup orphans: rc = -108 [ 3944.940980] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3946.715683] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3948.319424] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3953.677663] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3954.312647] Lustre: Permanently reactivating lustre-OST0000 [ 3960.301425] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3960.304313] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3960.307732] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3960.317617] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3960.322742] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3966.079925] Lustre: server umount lustre-OST0001 complete [ 3973.336455] Lustre: server umount lustre-OST0000 complete [ 3974.648875] Lustre: server umount lustre-MDT0000 complete [ 3975.747302] LustreError: 218619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772439635 with bad export cookie 9994661108443291899 [ 3975.751517] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3981.927049] Lustre: server umount lustre-MDT0001 complete [ 3985.009392] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 3985.898461] Key type lgssc unregistered [ 3986.018355] LNet: 223005:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3986.020131] LNetError: 223005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3986.030294] LNet: Removed LNI 192.168.201.141@tcp [ 3986.303392] Key type .llcrypt unregistered [ 3986.304345] Key type ._llcrypt unregistered [ 3994.175773] Key type ._llcrypt registered [ 3994.176734] Key type .llcrypt registered [ 3994.210393] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 3994.569387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3994.585589] alg: No test for adler32 (adler32-zlib) [ 3995.430735] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 3995.510076] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 3997.087169] Key type lgssc registered [ 3997.406467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4000.316403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4000.322329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4001.464980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4002.630251] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4005.250279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4005.285040] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4005.294500] Lustre: 224420:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4005.298883] Lustre: Permanently deactivating lustre-MDT0001 [ 4005.304892] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4005.413778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4006.593661] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4008.215204] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4009.275754] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4009.963786] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4009.966930] LustreError: 224460:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4009.969648] LustreError: 224460:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4009.972435] LustreError: 224460:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 1, failed: rc = -108 [ 4011.267244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4011.361505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4012.385045] LustreError: 226239:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4012.388691] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 4012.390172] LustreError: 226239:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4012.979736] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4014.684682] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4015.459850] LustreError: 226045:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4015.466692] LustreError: 226045:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4016.898579] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4016.973793] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4018.790420] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4019.127255] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 4019.148902] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 4020.520955] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4022.075483] Lustre: Permanently reactivating lustre-MDT0001 [ 4022.080726] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 4022.082580] Lustre: Skipped 2 previous similar messages [ 4028.709583] Lustre: lustre-MDT0001: Client 3aedb6de-ee13-4c7f-b34a-d7b6e752f6e9 (at 192.168.201.41@tcp) reconnecting [ 4030.568064] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4031.117082] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4032.760305] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 4032.833550] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 4034.178545] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 4034.261568] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 4035.128557] Lustre: Permanently deactivating lustre-MDT0001 [ 4035.133896] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4035.135918] Lustre: Skipped 2 previous similar messages [ 4044.798886] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4051.424680] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4051.427161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4051.432739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4052.960358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4052.962279] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4052.968390] Lustre: Skipped 2 previous similar messages [ 4054.232316] Lustre: server umount lustre-MDT0000 complete [ 4055.480242] LustreError: 224410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772439715 with bad export cookie 15751687869034358297 [ 4055.483530] LustreError: 224410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4055.484191] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4055.599567] Lustre: server umount lustre-MDT0001 complete [ 4062.972326] Lustre: server umount lustre-OST0000 complete [ 4066.640878] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 03:22:06 (1772439726) [ 4070.262721] Lustre: server umount lustre-OST0001 complete [ 4072.246082] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 4075.188300] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 4086.874445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4086.966521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4086.970142] Lustre: Skipped 2 previous similar messages [ 4086.978540] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4087.014190] Lustre: lustre-MDT0000: new disk, initializing [ 4087.038698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4087.045634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4088.304030] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4092.318662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4092.363493] Lustre: 232116:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4092.387371] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4092.389441] Lustre: Skipped 1 previous similar message [ 4092.428826] Lustre: lustre-MDT0001: new disk, initializing [ 4092.475666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4092.479154] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4093.813375] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4096.967750] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4098.104170] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4100.451950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4100.570632] Lustre: lustre-OST0000: new disk, initializing [ 4100.572871] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4100.600768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4100.604657] Lustre: Skipped 1 previous similar message [ 4102.267840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4102.272473] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4102.318387] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4102.575817] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4105.713880] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4106.352468] LustreError: 232122:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4110.499777] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4110.568810] Lustre: lustre-OST0001: new disk, initializing [ 4110.571140] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4112.535127] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4115.649089] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4116.199560] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4116.447187] LustreError: 232122:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4116.457356] LustreError: 232120:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4119.041849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4119.044607] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4119.076615] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4126.551097] LustreError: 232120:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4128.550763] Lustre: lustre-OST0001: Not available for connect from 192.168.201.41@tcp (stopping) [ 4128.553583] Lustre: Skipped 3 previous similar messages [ 4129.251591] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4129.251953] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4129.254183] LustreError: Skipped 1 previous similar message [ 4129.259356] Lustre: Skipped 1 previous similar message [ 4133.438737] Lustre: server umount lustre-OST0001 complete [ 4133.664083] LustreError: 233842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4139.492123] LustreError: 235737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4139.496397] LustreError: 235737:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4144.612166] LustreError: 235737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4144.617779] LustreError: 235737:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4148.707694] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4148.707821] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4148.711175] LustreError: Skipped 1 previous similar message [ 4148.711836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4148.715937] Lustre: Skipped 1 previous similar message [ 4148.717126] Lustre: Skipped 2 previous similar messages [ 4151.566502] Lustre: server umount lustre-OST0000 complete [ 4153.823882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4153.826491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4153.832221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4153.834357] Lustre: Skipped 3 previous similar messages [ 4165.088475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4165.091932] Lustre: Skipped 5 previous similar messages [ 4167.135178] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4167.203751] Lustre: server umount lustre-MDT0000 complete [ 4168.464268] LustreError: 234859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772439828 with bad export cookie 15751687869034360859 [ 4168.467889] LustreError: 234859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4168.467927] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4174.639458] Lustre: server umount lustre-MDT0001 complete [ 4178.328143] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 4179.297546] Key type lgssc unregistered [ 4179.419321] LNet: 236781:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4179.422811] LNetError: 236781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4179.431290] LNet: Removed LNI 192.168.201.141@tcp [ 4179.717605] Key type .llcrypt unregistered [ 4179.718637] Key type ._llcrypt unregistered [ 4194.170817] Key type ._llcrypt registered [ 4194.172278] Key type .llcrypt registered [ 4194.212239] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 4194.564583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4194.613164] alg: No test for adler32 (adler32-zlib) [ 4195.486813] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 4195.568707] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 4197.159172] Key type lgssc registered [ 4197.512648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4200.122542] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4200.128572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4201.251727] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4201.259220] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4201.317944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4202.576963] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4205.440286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4205.476052] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4205.485859] Lustre: 238950:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4205.579447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4206.822732] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4208.648949] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4209.891036] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4212.126140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4212.193447] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4212.256769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4214.090537] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4215.912851] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4218.381686] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 4264.416791] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4264.418083] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4264.418961] LustreError: Skipped 1 previous similar message [ 4264.423900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4264.426473] Lustre: Skipped 1 previous similar message [ 4265.943815] Lustre: server umount lustre-OST0000 complete [ 4267.434702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4271.491319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4271.613458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4273.386165] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4275.142800] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4284.900950] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4284.904431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4284.904657] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4284.910031] Lustre: Skipped 1 previous similar message [ 4310.498582] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4310.501381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4310.502702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4310.507702] Lustre: Skipped 2 previous similar messages [ 4314.579962] Lustre: server umount lustre-OST0000 complete [ 4318.175921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4318.178794] LustreError: Skipped 1 previous similar message [ 4318.180545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4318.186281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4318.189061] Lustre: Skipped 1 previous similar message [ 4320.736132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4320.736719] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4321.939396] Lustre: server umount lustre-MDT0000 complete [ 4323.109198] LustreError: 238942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772439983 with bad export cookie 9478271335237728398 [ 4323.113138] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4323.114561] LustreError: 238942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4323.223726] Lustre: server umount lustre-MDT0001 complete [ 4326.886535] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 4327.865662] Key type lgssc unregistered [ 4327.983473] LNet: 243900:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4327.985795] LNetError: 243900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4327.993388] LNet: Removed LNI 192.168.201.141@tcp [ 4328.327435] Key type .llcrypt unregistered [ 4328.328507] Key type ._llcrypt unregistered [ 4337.524235] Key type ._llcrypt registered [ 4337.525729] Key type .llcrypt registered [ 4337.565075] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 4338.015319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4338.033984] alg: No test for adler32 (adler32-zlib) [ 4338.881186] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 4338.966689] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 4340.543156] Key type lgssc registered [ 4340.899229] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4343.499392] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4343.504811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4344.685531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4346.035245] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4348.991871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4349.132215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4350.450754] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4352.247178] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4353.457305] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4355.659763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4355.785108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4357.585698] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4359.320300] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4369.130452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4379.618200] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4379.619252] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4379.619429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4379.626889] Lustre: Skipped 1 previous similar message [ 4383.116137] Lustre: server umount lustre-OST0000 complete [ 4384.736282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4384.741621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4384.745390] Lustre: Skipped 2 previous similar messages [ 4389.856416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4389.859489] Lustre: Skipped 1 previous similar message [ 4390.503300] Lustre: server umount lustre-MDT0000 complete [ 4391.724190] LustreError: 245256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772440051 with bad export cookie 13330138060330131743 [ 4391.727520] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4391.729719] LustreError: 245256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4391.841938] Lustre: server umount lustre-MDT0001 complete [ 4395.572945] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 4396.536659] Key type lgssc unregistered [ 4396.650382] LNet: 249247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4396.652703] LNetError: 249247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4396.661281] LNet: Removed LNI 192.168.201.141@tcp [ 4396.921024] Key type .llcrypt unregistered [ 4396.922479] Key type ._llcrypt unregistered [ 4403.929190] Key type ._llcrypt registered [ 4403.930924] Key type .llcrypt registered [ 4403.968978] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 4404.362486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4404.375858] alg: No test for adler32 (adler32-zlib) [ 4405.219312] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 4405.296093] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 4406.871214] Key type lgssc registered [ 4407.221322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4409.675874] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4409.681645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4410.839547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4412.077233] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4414.773157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4414.904151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4416.196764] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4417.906646] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4419.103286] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4421.335367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4421.481585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4423.285727] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4425.141141] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4435.818960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4445.151896] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4445.154171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4445.158139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4446.177698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4446.179128] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4446.181878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4450.449342] Lustre: server umount lustre-OST0000 complete [ 4455.903959] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4455.906149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4455.911130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4455.913236] Lustre: Skipped 1 previous similar message [ 4457.819218] Lustre: server umount lustre-MDT0000 complete [ 4459.024063] LustreError: 250411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772440119 with bad export cookie 5952321832282534004 [ 4459.027703] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4459.029728] LustreError: 250411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4459.140659] Lustre: server umount lustre-MDT0001 complete [ 4462.406521] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 4463.293368] Key type lgssc unregistered [ 4463.411323] LNet: 253774:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4463.413579] LNetError: 253774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4463.424274] LNet: Removed LNI 192.168.201.141@tcp [ 4463.681754] Key type .llcrypt unregistered [ 4463.682811] Key type ._llcrypt unregistered [ 4472.570267] Key type ._llcrypt registered [ 4472.571284] Key type .llcrypt registered [ 4472.615615] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 4472.986363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4472.996831] alg: No test for adler32 (adler32-zlib) [ 4473.847405] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 4473.923774] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 4475.503112] Key type lgssc registered [ 4475.847524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4478.387234] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4478.395614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4479.562545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4480.809160] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4483.642275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4483.761124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4485.066262] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4486.715978] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4487.863571] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4490.014178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4490.132024] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4491.179879] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4491.941803] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4493.611027] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4503.522475] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4503.522760] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4503.525534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4503.528271] Lustre: Skipped 1 previous similar message [ 4503.531113] Lustre: Skipped 1 previous similar message [ 4507.722120] Lustre: server umount lustre-OST0000 complete [ 4509.663901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4509.666164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4509.669939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4513.760217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4513.760888] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4515.031213] Lustre: server umount lustre-MDT0000 complete [ 4516.230203] LustreError: 255870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772440176 with bad export cookie 12323290513968811387 [ 4516.234195] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4516.234538] LustreError: 255870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4516.360619] Lustre: server umount lustre-MDT0001 complete [ 4520.031155] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 4521.003984] Key type lgssc unregistered [ 4521.127420] LNet: 259099:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4521.131113] LNetError: 259099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4521.138311] LNet: Removed LNI 192.168.201.141@tcp [ 4521.442319] Key type .llcrypt unregistered [ 4521.444056] Key type ._llcrypt unregistered [ 4528.392398] Key type ._llcrypt registered [ 4528.394024] Key type .llcrypt registered [ 4528.438035] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 4528.810174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4528.852051] alg: No test for adler32 (adler32-zlib) [ 4529.717961] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 4529.803512] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 4531.383157] Key type lgssc registered [ 4531.728755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4534.227134] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4534.233958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4535.414650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4536.723189] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4539.539946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4539.667833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4540.961847] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4542.687069] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4543.809041] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4546.100143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4546.210429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4548.017929] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4549.748241] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4561.578381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4570.080054] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4570.083707] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4570.089968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4572.131209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4572.131571] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4572.132815] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4574.933934] Lustre: server umount lustre-OST0000 complete [ 4577.248747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4577.249731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4577.253031] Lustre: Skipped 1 previous similar message [ 4577.255838] Lustre: Skipped 1 previous similar message [ 4582.368141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4582.370300] Lustre: Skipped 2 previous similar messages [ 4582.438209] Lustre: server umount lustre-MDT0000 complete [ 4583.717075] LustreError: 260260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772440243 with bad export cookie 16574721636644506693 [ 4583.720326] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4583.721441] LustreError: 260260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4583.840106] Lustre: server umount lustre-MDT0001 complete [ 4587.440411] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 4588.479950] Key type lgssc unregistered [ 4588.611548] LNet: 263659:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4588.613725] LNetError: 263659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4588.620439] LNet: Removed LNI 192.168.201.141@tcp [ 4588.949116] Key type .llcrypt unregistered [ 4588.950138] Key type ._llcrypt unregistered [ 4592.630914] Key type ._llcrypt registered [ 4592.631952] Key type .llcrypt registered [ 4592.667989] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4598.045578] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 4603.994094] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 4604.303856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4604.350358] alg: No test for adler32 (adler32-zlib) [ 4605.204712] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 4605.286961] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 4606.863153] Key type lgssc registered [ 4607.186605] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4618.120508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4618.126695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4619.213842] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4619.226763] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4619.272943] Lustre: lustre-MDT0000: new disk, initializing [ 4619.297690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4619.304470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4620.585530] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4624.639109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4624.685544] Lustre: 267131:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4624.713631] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4624.716027] Lustre: Skipped 1 previous similar message [ 4624.757557] Lustre: lustre-MDT0001: new disk, initializing [ 4624.794435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4624.806109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4624.809295] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4626.152386] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4629.210088] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4630.476707] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4632.724584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4632.842045] Lustre: lustre-OST0000: new disk, initializing [ 4632.843928] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4632.876773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4634.747899] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4637.180272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4637.183799] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4637.211655] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4637.787193] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4639.121230] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4639.195089] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4640.462844] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4640.535469] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4642.273114] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4642.273587] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4642.275818] LustreError: Skipped 1 previous similar message [ 4642.280091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4642.283431] Lustre: Skipped 1 previous similar message [ 4643.219332] Lustre: server umount lustre-OST0000 complete [ 4645.343828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4645.346120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4645.349374] Lustre: Skipped 1 previous similar message [ 4645.351260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4647.393078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4647.394095] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4650.590368] Lustre: server umount lustre-MDT0000 complete [ 4651.872443] LustreError: 267123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772440311 with bad export cookie 8176418913682196037 [ 4651.876054] LustreError: 267123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4651.876241] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4651.995887] Lustre: server umount lustre-MDT0001 complete [ 4654.789383] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 03:31:54 (1772440314) [ 4656.973873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4657.175356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4658.462302] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4661.319339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4661.430464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4662.630150] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4664.404406] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4665.645962] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4667.961177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4669.993286] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4671.862753] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4686.815853] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4686.819737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4686.824868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4686.827814] Lustre: Skipped 1 previous similar message [ 4691.344184] Lustre: server umount lustre-OST0000 complete [ 4692.960404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4692.963354] Lustre: Skipped 1 previous similar message [ 4692.964851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4692.966692] Lustre: Skipped 2 previous similar messages [ 4696.033265] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 4696.035966] LustreError: Skipped 1 previous similar message [ 4698.772854] Lustre: server umount lustre-MDT0000 complete [ 4699.917342] LustreError: 271117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772440360 with bad export cookie 8176418913682196912 [ 4699.919684] LustreError: 271854:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4699.920194] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4699.920864] LustreError: 271117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4700.045980] Lustre: server umount lustre-MDT0001 complete [ 4703.385551] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 4704.279676] Key type lgssc unregistered [ 4704.409357] LNet: 274333:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4704.411577] LNetError: 274333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4704.420261] LNet: Removed LNI 192.168.201.141@tcp [ 4704.689891] Key type .llcrypt unregistered [ 4704.691039] Key type ._llcrypt unregistered [ 4713.021441] Key type ._llcrypt registered [ 4713.022537] Key type .llcrypt registered [ 4713.063646] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 4719.021728] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 4719.396447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4719.434070] alg: No test for adler32 (adler32-zlib) [ 4720.296793] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 4720.380644] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 4721.951206] Key type lgssc registered [ 4722.312925] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4736.608435] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4736.613440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4737.704215] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4737.715371] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4737.750152] Lustre: lustre-MDT0000: new disk, initializing [ 4737.775049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4737.781665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4739.045669] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4743.011136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4743.050733] Lustre: 278097:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4743.071744] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4743.073331] Lustre: Skipped 1 previous similar message [ 4743.113221] Lustre: lustre-MDT0001: new disk, initializing [ 4743.143063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4743.153882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4743.156747] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4744.347225] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4747.387291] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4748.534799] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4750.772415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4750.879328] Lustre: lustre-OST2710: new disk, initializing [ 4750.881172] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4750.911598] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4752.941496] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4753.910059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4753.913964] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4753.943254] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4756.082733] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4758.405844] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4758.484725] Lustre: lustre-OST03e8: new disk, initializing [ 4758.486838] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4758.518425] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4759.679967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4759.684703] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4759.718907] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4760.536217] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4762.344300] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4765.090932] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4765.185377] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4766.659623] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4766.736522] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4768.062901] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4768.140443] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4769.542994] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4769.623808] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4773.855815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4773.858067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4773.861895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4775.394516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4775.395298] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4775.396580] Lustre: Skipped 1 previous similar message [ 4775.400342] Lustre: Skipped 2 previous similar messages [ 4776.795064] Lustre: server umount lustre-MDT0000 complete [ 4778.004148] LustreError: 278874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772440438 with bad export cookie 6610828056966744385 [ 4778.008386] LustreError: 278874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4778.008992] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4778.127841] Lustre: server umount lustre-MDT0001 complete [ 4789.699477] Lustre: server umount lustre-OST2710 complete [ 4800.162037] Lustre: server umount lustre-OST03e8 complete [ 4806.098918] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 4809.224706] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 4820.862680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4820.949334] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4820.959315] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4820.996384] Lustre: lustre-MDT0000: new disk, initializing [ 4821.019385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4821.025181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4822.256139] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4826.361167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4826.400413] Lustre: 285813:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4826.492313] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4827.688816] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4830.748713] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4831.895940] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4834.196868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4834.301077] Lustre: lustre-OST0000: new disk, initializing [ 4834.302532] Lustre: Skipped 1 previous similar message [ 4834.304061] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4834.305569] Lustre: Skipped 2 previous similar messages [ 4834.332225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4834.335557] Lustre: Skipped 1 previous similar message [ 4836.166384] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4837.558019] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4837.560882] Lustre: Skipped 1 previous similar message [ 4837.562843] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4837.588230] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4839.179738] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4840.603983] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4840.677968] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4841.967814] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4842.041350] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4842.977684] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4842.977812] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4842.984302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4842.986019] Lustre: Skipped 2 previous similar messages [ 4848.098675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4848.101195] Lustre: Skipped 1 previous similar message [ 4848.784721] Lustre: server umount lustre-OST0000 complete [ 4852.191755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4852.193849] LustreError: Skipped 1 previous similar message [ 4852.195119] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4852.198159] Lustre: Skipped 1 previous similar message [ 4852.199791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4856.215113] Lustre: server umount lustre-MDT0000 complete [ 4857.410276] LustreError: 287557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772440517 with bad export cookie 6610828056966746051 [ 4857.414067] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4857.414071] LustreError: 287557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4857.524303] Lustre: server umount lustre-MDT0001 complete [ 4860.323371] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 03:35:19 (1772440519) [ 4860.825471] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4861.402630] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 03:35:20 (1772440520) [ 4862.391503] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 4871.239833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4871.338502] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4871.346440] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4871.403927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4872.603760] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4875.378036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4875.413907] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4875.422802] Lustre: 291245:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4876.810778] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4878.629980] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4879.774140] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4892.962748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4893.033157] Lustre: Denying initial registration attempt from nid 192.168.201.141@tcp, specified as failover [ 4893.036269] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4893.038698] LustreError: 292851:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -99 [ 4893.040859] LustreError: 292851:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 4893.042537] LustreError: 292851:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4893.063768] Lustre: server umount lustre-OST0000 complete [ 4893.065031] LustreError: 292851:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4896.223750] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4896.225946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4896.229231] Lustre: Skipped 1 previous similar message [ 4896.230697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4896.232415] Lustre: Skipped 2 previous similar messages [ 4901.268469] Lustre: server umount lustre-MDT0000 complete [ 4902.451266] LustreError: 291235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772440562 with bad export cookie 6610828056966746919 [ 4902.453892] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4902.454583] LustreError: 291235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4902.564700] Lustre: server umount lustre-MDT0001 complete [ 4906.093596] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 4907.050399] Key type lgssc unregistered [ 4907.171361] LNet: 293955:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4907.173821] LNetError: 293955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4907.182393] LNet: Removed LNI 192.168.201.141@tcp [ 4907.462743] Key type .llcrypt unregistered [ 4907.463776] Key type ._llcrypt unregistered [ 4911.041990] Key type ._llcrypt registered [ 4911.043277] Key type .llcrypt registered [ 4911.080685] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 4911.512820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4911.533947] alg: No test for adler32 (adler32-zlib) [ 4912.379643] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 4912.463483] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 4914.039122] Key type lgssc registered [ 4914.374675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4949.201926] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 4952.195273] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4952.202927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4953.309691] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4953.320603] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4953.382491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4954.753527] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4957.611351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4957.654632] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4957.667660] Lustre: 296668:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4957.771136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4959.120778] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4960.976407] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4962.273140] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4975.628554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4975.705336] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4975.778716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4977.776667] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4979.696067] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4980.450777] Lustre: server umount lustre-OST0000 complete [ 4983.264665] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4983.267873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4983.271932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4986.336114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4986.336732] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4987.933571] Lustre: server umount lustre-MDT0000 complete [ 4989.262698] LustreError: 296659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772440649 with bad export cookie 18266191353987587819 [ 4989.265709] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4989.267880] LustreError: 296659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4989.400702] Lustre: server umount lustre-MDT0001 complete [ 4993.078169] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 4994.102236] Key type lgssc unregistered [ 4994.224463] LNet: 299876:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4994.228212] LNetError: 299876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4994.238344] LNet: Removed LNI 192.168.201.141@tcp [ 4994.555504] Key type .llcrypt unregistered [ 4994.557141] Key type ._llcrypt unregistered [ 5003.764707] Key type ._llcrypt registered [ 5003.765776] Key type .llcrypt registered [ 5003.801073] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 5004.124937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5004.167595] alg: No test for adler32 (adler32-zlib) [ 5005.014880] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 5005.096316] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 5006.671164] Key type lgssc registered [ 5007.002745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5009.421648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5009.429575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5010.616363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5011.904817] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5014.738841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5014.870563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5016.114502] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5017.933363] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5019.218430] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5021.413845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5021.535439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5023.314214] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5025.104029] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5045.727722] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5045.729953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5045.734450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5046.241712] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5046.241825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5046.245299] Lustre: Skipped 1 previous similar message [ 5046.247218] Lustre: Skipped 2 previous similar messages [ 5049.107145] Lustre: server umount lustre-MDT0000 complete [ 5050.316200] LustreError: 301232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772440710 with bad export cookie 16937618314339357981 [ 5050.319553] LustreError: 301232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5050.319739] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5051.166390] LustreError: 303408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5051.172853] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.41@tcp (stopping) [ 5051.174819] LustreError: 303408:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5051.360299] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5055.392125] LustreError: 301248:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5055.392468] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5055.396354] LustreError: 301248:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5055.402725] Lustre: Skipped 1 previous similar message [ 5056.500495] Lustre: server umount lustre-MDT0001 complete [ 5058.590025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5061.280139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5075.935670] LustreError: 300562:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000bddc2ad5 x1858538742768768/t0(0) o250->MGC192.168.201.141@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5076.036354] LustreError: 302856:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5076.044702] LustreError: 302856:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5076.060345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5076.077625] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 5077.282379] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5080.118304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5080.161163] LustreError: 304531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5080.165806] LustreError: 304531:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5080.246592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5081.567657] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5083.382656] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5085.347996] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5088.018037] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5089.121153] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5089.636143] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5091.295746] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5091.297974] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5091.301683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5091.303544] Lustre: Skipped 1 previous similar message [ 5095.906241] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5095.909299] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5101.025752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5101.028130] Lustre: Skipped 3 previous similar messages [ 5105.119112] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5105.173233] Lustre: server umount lustre-OST0000 complete [ 5111.264762] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5111.265071] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5111.271258] Lustre: Skipped 1 previous similar message [ 5112.544827] Lustre: server umount lustre-MDT0000 complete [ 5113.759273] LustreError: 301234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772440773 with bad export cookie 16937618314339365135 [ 5113.762693] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5113.764238] LustreError: 301234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5113.878223] Lustre: server umount lustre-MDT0001 complete [ 5117.346211] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 5118.358747] Key type lgssc unregistered [ 5118.483412] LNet: 306963:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5118.486520] LNetError: 306963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5118.495306] LNet: Removed LNI 192.168.201.141@tcp [ 5118.784596] Key type .llcrypt unregistered [ 5118.785636] Key type ._llcrypt unregistered [ 5127.302991] Key type ._llcrypt registered [ 5127.304243] Key type .llcrypt registered [ 5127.347849] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 5127.765042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5127.773593] alg: No test for adler32 (adler32-zlib) [ 5128.611268] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 5128.686594] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 5130.263205] Key type lgssc registered [ 5130.663099] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5133.176405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5133.182212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5134.332869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5135.560557] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5138.330560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5138.455759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5139.760416] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5141.539506] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5142.742643] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5148.641671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5148.642242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5148.642981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5148.650602] Lustre: Skipped 1 previous similar message [ 5149.664329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5149.666223] Lustre: Skipped 2 previous similar messages [ 5154.784052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5154.785819] Lustre: Skipped 1 previous similar message [ 5158.367109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5158.427034] Lustre: server umount lustre-MDT0000 complete [ 5159.654284] LustreError: 308321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772440819 with bad export cookie 12412049017722783424 [ 5159.657021] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5159.657400] LustreError: 308321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5159.764917] Lustre: server umount lustre-MDT0001 complete [ 5164.554451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5164.654637] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5164.666898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5164.738150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5166.061056] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5168.861913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5168.892876] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5168.902512] Lustre: 310701:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5168.997692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5170.280902] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5172.127350] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5173.448484] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5176.052608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5176.133088] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5176.136352] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5176.139901] LustreError: 312353:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -2 [ 5176.142326] LustreError: 312353:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 5176.144031] LustreError: 312353:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 5176.171075] Lustre: server umount lustre-OST0000 complete [ 5176.173394] LustreError: 312353:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5179.097178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5179.165427] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5179.245132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5181.222756] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 5181.251977] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5183.118219] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5185.793352] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5185.828132] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5185.863385] Lustre: lustre-OST0001: new disk, initializing [ 5185.865228] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5187.671106] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5188.366630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5188.371425] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5188.403816] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5189.393145] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5193.704298] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5193.704392] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5193.709529] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5193.711268] LustreError: Skipped 1 previous similar message [ 5196.419818] Lustre: server umount lustre-OST0001 complete [ 5198.818023] LustreError: 313785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5198.818337] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5198.820474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5198.820479] Lustre: Skipped 1 previous similar message [ 5198.824238] LustreError: 313785:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5198.829238] Lustre: Skipped 2 previous similar messages [ 5203.791427] Lustre: server umount lustre-OST0000 complete [ 5209.056833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5209.057331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5209.060454] Lustre: Skipped 1 previous similar message [ 5209.065912] Lustre: Skipped 2 previous similar messages [ 5211.225620] Lustre: server umount lustre-MDT0000 complete [ 5212.449247] LustreError: 310736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772440872 with bad export cookie 12412049017722783963 [ 5212.453078] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5212.454054] LustreError: 310736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5212.579255] Lustre: server umount lustre-MDT0001 complete [ 5216.086152] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 5217.198668] Key type lgssc unregistered [ 5217.337340] LNet: 315604:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5217.339567] LNetError: 315604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5217.347426] LNet: Removed LNI 192.168.201.141@tcp [ 5217.634081] Key type .llcrypt unregistered [ 5217.635118] Key type ._llcrypt unregistered [ 5247.601977] Key type ._llcrypt registered [ 5247.602934] Key type .llcrypt registered [ 5247.640954] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 5253.715370] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 5254.146292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5254.153607] alg: No test for adler32 (adler32-zlib) [ 5255.011400] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 5255.100686] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 5256.687185] Key type lgssc registered [ 5257.080567] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5268.217249] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5268.223325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5269.331110] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5269.343670] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5269.387596] Lustre: lustre-MDT0000: new disk, initializing [ 5269.417287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5269.424426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5270.633197] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5274.749272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5274.792645] Lustre: 320471:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5274.817370] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5274.819175] Lustre: Skipped 1 previous similar message [ 5274.858174] Lustre: lustre-MDT0001: new disk, initializing [ 5274.890762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5274.904179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5274.907407] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5276.147302] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5279.241455] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5280.435177] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5282.725334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5282.839225] Lustre: lustre-OST0000: new disk, initializing [ 5282.841473] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5282.870688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5284.796446] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5287.866950] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5289.220757] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5289.977844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5289.982102] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5290.011241] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5290.314789] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5291.681113] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5291.751665] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5295.074534] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5295.074780] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5295.080907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5295.082777] Lustre: Skipped 1 previous similar message [ 5298.510026] Lustre: server umount lustre-OST0000 complete [ 5300.192862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5300.193726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5300.197646] Lustre: Skipped 2 previous similar messages [ 5305.312626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5305.315640] Lustre: Skipped 2 previous similar messages [ 5310.432529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5310.435528] Lustre: Skipped 1 previous similar message [ 5314.015091] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5314.085775] Lustre: server umount lustre-MDT0000 complete [ 5315.324225] LustreError: 320462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772440975 with bad export cookie 1326395174295480421 [ 5315.327368] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5315.329100] LustreError: 320462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5315.440980] Lustre: server umount lustre-MDT0001 complete [ 5318.231683] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 03:42:57 (1772440977) [ 5320.528286] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 03:43:00 (1772440980) [ 5323.247269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5323.449029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5324.742274] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5327.526256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5327.647250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5328.950082] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5330.691388] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5331.850031] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5334.121030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5336.015593] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5337.719460] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5356.531193] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5357.082561] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5357.580163] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5358.123944] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5358.757606] Lustre: Failing over lustre-MDT0000 [ 5358.908912] Lustre: server umount lustre-MDT0000 complete [ 5360.096891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5360.098597] LustreError: 324818:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5360.100614] Lustre: Skipped 2 previous similar messages [ 5360.106549] LustreError: 324818:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5363.493121] LustreError: 324818:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5365.216809] LustreError: 324820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5365.222933] LustreError: 324820:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5368.607040] LustreError: 326974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5371.803444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5371.844574] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5371.958431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5371.961553] Lustre: Skipped 1 previous similar message [ 5372.003592] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5373.317969] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5373.726591] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5377.000856] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5377.009306] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5377.026621] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5378.193451] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5378.712338] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5380.559965] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5381.201475] Lustre: Failing over lustre-MDT0000 [ 5381.479685] Lustre: server umount lustre-MDT0000 complete [ 5382.114779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5382.114780] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5382.116023] LustreError: 324820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5382.116911] LustreError: Skipped 1 previous similar message [ 5382.123413] LustreError: 324820:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5392.354102] LustreError: 324823:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5392.360909] LustreError: 324823:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 5394.325137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5394.365153] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5394.460670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5394.483877] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5395.708487] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5399.327209] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5399.527636] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5399.529444] Lustre: Skipped 2 previous similar messages [ 5399.536687] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5399.553359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5400.673825] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5401.211528] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5404.640932] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5404.641895] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5404.646461] Lustre: Skipped 2 previous similar messages [ 5404.648465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5404.650065] Lustre: Skipped 2 previous similar messages [ 5409.621663] Lustre: server umount lustre-MDT0000 complete [ 5409.760663] LustreError: 324824:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5409.765035] LustreError: 324824:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5410.845214] LustreError: 326436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772441070 with bad export cookie 1326395174295483102 [ 5410.848586] LustreError: 326436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5410.848931] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5410.970336] Lustre: server umount lustre-MDT0001 complete [ 5422.489036] Lustre: server umount lustre-OST0000 complete [ 5427.975988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5428.157475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5429.591273] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5432.464973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5433.920853] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5435.771541] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5436.926984] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5439.185044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5441.219251] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5443.052929] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5454.694669] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5463.008367] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5463.015082] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5463.022175] Lustre: Skipped 2 previous similar messages [ 5463.024905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5474.271125] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5474.327114] Lustre: server umount lustre-OST0000 complete [ 5478.879789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5478.882075] LustreError: Skipped 1 previous similar message [ 5480.416647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5480.417536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5480.418716] Lustre: Skipped 5 previous similar messages [ 5480.423176] Lustre: Skipped 2 previous similar messages [ 5481.699529] Lustre: server umount lustre-MDT0000 complete [ 5482.993496] LustreError: 330598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772441143 with bad export cookie 1326395174295483823 [ 5482.997534] LustreError: 330598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5482.998833] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5483.119913] Lustre: server umount lustre-MDT0001 complete [ 5486.915478] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 5487.934141] Key type lgssc unregistered [ 5488.058360] LNet: 333819:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5488.060799] LNetError: 333819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5488.070242] LNet: Removed LNI 192.168.201.141@tcp [ 5488.358444] Key type .llcrypt unregistered [ 5488.359615] Key type ._llcrypt unregistered [ 5498.162028] Key type ._llcrypt registered [ 5498.163048] Key type .llcrypt registered [ 5498.197370] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 5498.519207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5498.566530] alg: No test for adler32 (adler32-zlib) [ 5499.415794] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 5499.502927] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 5501.079111] Key type lgssc registered [ 5501.511131] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5504.042470] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5504.048197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5505.205276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5506.447374] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5509.275317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5509.403888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5510.762922] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5512.548538] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5513.889654] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5516.112613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5516.233874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5518.106648] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5519.890686] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5530.602932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5540.319826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5540.322394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5540.329885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5540.835173] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5540.836979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5540.841405] Lustre: Skipped 1 previous similar message [ 5540.848625] Lustre: Skipped 1 previous similar message [ 5545.948603] Lustre: server umount lustre-MDT0000 complete [ 5545.955673] LustreError: 337405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5545.960577] LustreError: 337405:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5547.205523] LustreError: 335285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772441207 with bad export cookie 16029522675660708430 [ 5547.208160] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5547.208663] LustreError: 335285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5547.327082] Lustre: server umount lustre-MDT0001 complete [ 5558.817182] Lustre: server umount lustre-OST0000 complete [ 5563.615945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5563.780274] LustreError: 338695:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5563.805703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5565.077501] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5567.844802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5567.969616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5569.410522] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5571.166433] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5572.366157] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5574.615057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5576.604680] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5578.759574] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5588.072334] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5598.690491] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5598.692258] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5598.693339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5598.699179] Lustre: Skipped 2 previous similar messages [ 5602.270305] Lustre: server umount lustre-MDT0000 complete [ 5603.473438] LustreError: 340974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772441263 with bad export cookie 16029522675660709529 [ 5603.476390] LustreError: 340974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5603.476910] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5603.595052] Lustre: server umount lustre-MDT0001 complete [ 5615.261161] Lustre: server umount lustre-OST0000 complete [ 5620.185164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5620.324508] LustreError: 342288:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5620.347363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5620.350295] Lustre: Skipped 1 previous similar message [ 5621.546683] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5624.225271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5625.581176] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5627.376617] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5628.607726] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5630.907823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5633.053782] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5634.894670] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5645.349902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5649.376464] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5649.381445] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5649.389827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5649.393733] Lustre: Skipped 2 previous similar messages [ 5653.215087] Lustre: server umount lustre-OST0000 complete [ 5655.007873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5655.011568] LustreError: Skipped 1 previous similar message [ 5655.014207] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5655.019453] Lustre: Skipped 1 previous similar message [ 5655.021559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5655.024424] Lustre: Skipped 2 previous similar messages [ 5665.760774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5665.764289] Lustre: Skipped 4 previous similar messages [ 5669.343102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5669.402335] Lustre: server umount lustre-MDT0000 complete [ 5670.668430] LustreError: 342268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772441330 with bad export cookie 16029522675660710537 [ 5670.671239] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5670.672049] LustreError: 342268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5670.780453] Lustre: server umount lustre-MDT0001 complete [ 5674.643778] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 5675.617390] Key type lgssc unregistered [ 5675.744374] LNet: 345491:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5675.746679] LNetError: 345491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5675.754316] LNet: Removed LNI 192.168.201.141@tcp [ 5676.064618] Key type .llcrypt unregistered [ 5676.065978] Key type ._llcrypt unregistered [ 5686.842460] Key type ._llcrypt registered [ 5686.843659] Key type .llcrypt registered [ 5686.884114] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 5687.329699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5687.338056] alg: No test for adler32 (adler32-zlib) [ 5688.194020] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 5688.273550] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 5689.855196] Key type lgssc registered [ 5690.284540] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5692.891056] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5692.899378] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5692.902397] LustreError: 346890:0:(osd_handler.c:8531:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5692.911236] LustreError: 346890:0:(obd_config.c:880:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5692.913663] LustreError: 346890:0:(obd_mount.c:217:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5692.916131] LustreError: 346890:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5692.918857] LustreError: 346890:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5695.761649] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5695.766982] LustreError: 347287:0:(osd_handler.c:8531:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5695.778213] LustreError: 347287:0:(obd_config.c:880:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5695.781060] LustreError: 347287:0:(obd_mount.c:217:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5695.784853] LustreError: 347287:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5695.789079] LustreError: 347287:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5702.899459] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 5703.994164] Key type lgssc unregistered [ 5704.137375] LNet: 348271:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5704.139415] LNetError: 348271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5704.152292] LNet: Removed LNI 192.168.201.141@tcp [ 5704.411425] Key type .llcrypt unregistered [ 5704.412606] Key type ._llcrypt unregistered [ 5710.438764] Key type ._llcrypt registered [ 5710.439833] Key type .llcrypt registered [ 5710.475271] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 5717.279875] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 5717.665219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5717.673831] alg: No test for adler32 (adler32-zlib) [ 5718.511030] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 5718.586450] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 5720.159184] Key type lgssc registered [ 5720.718069] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5732.495765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5732.501575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5733.585725] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5733.597871] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5733.642938] Lustre: lustre-MDT0000: new disk, initializing [ 5733.672395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5733.681750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5734.929202] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5739.577945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5739.649614] Lustre: 351363:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5739.686540] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5739.689037] Lustre: Skipped 1 previous similar message [ 5739.731014] Lustre: lustre-MDT0001: new disk, initializing [ 5739.769570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5739.783413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5739.787648] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5741.182182] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5744.237412] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5745.543039] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5747.947626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5748.077611] Lustre: lustre-OST0000: new disk, initializing [ 5748.080378] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5748.123178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5750.322680] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5750.402893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5750.406269] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5750.454410] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5753.721602] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5755.134558] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5755.209242] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5756.605969] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5756.685552] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5760.993194] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5760.993834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5760.995127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5760.995221] LustreError: Skipped 1 previous similar message [ 5760.998308] Lustre: Skipped 1 previous similar message [ 5763.477985] Lustre: server umount lustre-OST0000 complete [ 5765.599861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5765.601836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5765.605380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5765.606820] Lustre: Skipped 1 previous similar message [ 5770.994739] Lustre: server umount lustre-MDT0000 complete [ 5772.343183] LustreError: 351357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772441432 with bad export cookie 17571107902719835700 [ 5772.346256] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5772.346682] LustreError: 351357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5772.463620] Lustre: server umount lustre-MDT0001 complete [ 5775.587910] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5776.157645] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 03:50:35 (1772441435) [ 5778.278442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5778.458967] LustreError: 355417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5778.467465] LustreError: 355417:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5778.489479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5779.776719] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5782.553139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5782.667156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5784.082539] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5785.945327] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5787.106171] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5789.508940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5791.751820] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5793.469608] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5795.919367] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5795.991280] Lustre: lustre-OST0001: new disk, initializing [ 5795.994128] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5796.030636] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5796.032386] Lustre: Skipped 1 previous similar message [ 5797.801227] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5801.061455] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5802.494444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5802.497459] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5802.527185] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5807.585302] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5807.585352] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5807.592552] LustreError: Skipped 1 previous similar message [ 5807.599842] Lustre: Skipped 1 previous similar message [ 5807.605811] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5807.613744] Lustre: Skipped 3 previous similar messages [ 5809.218850] Lustre: server umount lustre-OST0001 complete [ 5809.635345] LustreError: 357012:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5809.639494] LustreError: 357012:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5814.757959] LustreError: 358523:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5814.763556] LustreError: 358523:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5819.878770] LustreError: 357010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5819.884363] LustreError: 357010:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5823.456150] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5823.458989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5823.464633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5824.994973] LustreError: 357010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5824.999877] LustreError: 357010:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5827.286636] Lustre: server umount lustre-OST0000 complete [ 5829.087984] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5829.089952] LustreError: Skipped 1 previous similar message [ 5829.091085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5829.094012] Lustre: Skipped 1 previous similar message [ 5829.095525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5829.097039] Lustre: Skipped 2 previous similar messages [ 5834.784556] Lustre: server umount lustre-MDT0000 complete [ 5835.232301] LustreError: 355436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5835.242537] LustreError: 355436:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5836.246134] LustreError: 355399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772441496 with bad export cookie 17571107902719836568 [ 5836.255547] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5842.414531] Lustre: server umount lustre-MDT0001 complete [ 5845.945141] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 5847.112631] Key type lgssc unregistered [ 5847.259597] LNet: 359768:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5847.263713] LNetError: 359768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5847.274490] LNet: Removed LNI 192.168.201.141@tcp [ 5847.606378] Key type .llcrypt unregistered [ 5847.608396] Key type ._llcrypt unregistered [ 5865.673597] Key type ._llcrypt registered [ 5865.674697] Key type .llcrypt registered [ 5865.726611] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 5866.322489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5866.348119] alg: No test for adler32 (adler32-zlib) [ 5867.226958] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 5867.331259] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 5868.919151] Key type lgssc registered [ 5869.257063] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5871.858934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5871.865927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5872.960349] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5872.970064] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5873.033421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5874.226355] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5877.230172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5877.266818] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5877.276380] Lustre: 362524:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5877.370453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5878.860165] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5880.692499] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5881.959791] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5882.849141] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5882.852044] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5882.857453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5884.384195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5884.384641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5889.504050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5889.506064] Lustre: Skipped 2 previous similar messages [ 5894.624124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5894.625941] Lustre: Skipped 1 previous similar message [ 5897.183102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5897.259047] Lustre: server umount lustre-MDT0000 complete [ 5898.590051] LustreError: 362514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772441558 with bad export cookie 13748553900004903148 [ 5898.593091] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5898.593340] LustreError: 362514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5898.704864] Lustre: server umount lustre-MDT0001 complete [ 5902.377759] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 03:52:41 (1772441561) [ 5904.974268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5905.149089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5906.614424] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5909.752386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5909.865534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5911.143704] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5913.760621] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5915.138541] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5917.911455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5917.989712] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5918.065216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5919.939366] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5921.529537] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5930.251637] Lustre: Permanently deactivating lustre-OST0000 [ 5930.256040] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5933.420953] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5934.989108] LustreError: 367407:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5934.994466] LustreError: 367407:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5934.997832] LustreError: 367407:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5935.577376] LustreError: 367455:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5935.579564] LustreError: 367455:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5942.371285] Lustre: server umount lustre-OST0000 complete [ 5943.292748] LustreError: 367704:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5943.296601] LustreError: 367704:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5945.825762] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5945.826935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5945.827771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5945.844635] Lustre: Skipped 1 previous similar message [ 5954.047406] Lustre: server umount lustre-MDT0000 complete [ 5955.644859] LustreError: 364791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772441615 with bad export cookie 13748553900004903708 [ 5955.650928] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5955.651103] LustreError: 364791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5965.852813] Lustre: server umount lustre-MDT0001 complete [ 5969.209950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5970.878254] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5972.308349] LustreError: 368844:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5975.178737] Lustre: 369183:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.141@tcp [ 5975.863940] Lustre: server umount MGS complete [ 5979.569605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5979.728717] LustreError: 369906:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5979.733063] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5979.733873] LustreError: 369906:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5979.736867] Lustre: Skipped 1 previous similar message [ 5979.759139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5980.914535] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5984.127905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5984.255286] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5985.815530] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5987.486705] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5988.851061] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5989.444903] Lustre: Permanently reactivating lustre-OST0000 [ 5989.449460] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5989.451236] Lustre: Skipped 2 previous similar messages [ 6015.967123] LustreError: 371507:0:(osp_dev.c:694:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 6020.058894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6020.240970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6020.244853] Lustre: Skipped 1 previous similar message [ 6022.537772] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6024.305789] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6036.964594] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6036.965001] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6036.967270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6036.967277] Lustre: Skipped 3 previous similar messages [ 6036.973439] Lustre: Skipped 1 previous similar message [ 6038.092499] Lustre: server umount lustre-OST0000 complete [ 6040.544652] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6041.388259] Lustre: server umount lustre-MDT0000 complete [ 6042.081166] LustreError: 369906:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6042.081639] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6042.088487] Lustre: Skipped 1 previous similar message [ 6042.639364] LustreError: 369888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772441702 with bad export cookie 13748553900004904751 [ 6042.642695] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6042.646067] LustreError: 369888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6042.768723] Lustre: server umount lustre-MDT0001 complete [ 6046.579675] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 6047.679509] Key type lgssc unregistered [ 6047.802294] LNet: 374226:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6047.804581] LNetError: 374226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6047.810386] LNet: Removed LNI 192.168.201.141@tcp [ 6048.075319] Key type .llcrypt unregistered [ 6048.076399] Key type ._llcrypt unregistered [ 6054.351945] Key type ._llcrypt registered [ 6054.353152] Key type .llcrypt registered [ 6054.385640] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 6062.027109] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 6062.343514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6062.374912] alg: No test for adler32 (adler32-zlib) [ 6063.220882] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 6063.296938] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 6064.879147] Key type lgssc registered [ 6065.226404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6078.872434] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6078.878789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6079.982155] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6079.993470] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6080.036718] Lustre: lustre-MDT0000: new disk, initializing [ 6080.061841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6080.069204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6081.447479] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6085.752049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6085.792803] Lustre: 377322:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6085.812301] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6085.814191] Lustre: Skipped 1 previous similar message [ 6085.849267] Lustre: lustre-MDT0001: new disk, initializing [ 6085.880903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6085.892423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6085.897564] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6087.231566] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6090.343988] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6091.376915] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6093.467579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6093.576022] Lustre: lustre-OST0000: new disk, initializing [ 6093.577962] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6093.604082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6095.391506] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6095.415772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6095.418663] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6095.452407] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6098.470281] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6099.794478] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6099.888930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6101.523350] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6101.629416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6105.568714] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6105.571097] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6105.571592] LustreError: Skipped 1 previous similar message [ 6105.578044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6105.580987] Lustre: Skipped 1 previous similar message [ 6108.569001] Lustre: server umount lustre-OST0000 complete [ 6110.689328] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6110.690330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6110.695352] Lustre: Skipped 2 previous similar messages [ 6110.699596] Lustre: Skipped 1 previous similar message [ 6115.808215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6115.810525] Lustre: Skipped 1 previous similar message [ 6115.924619] Lustre: server umount lustre-MDT0000 complete [ 6117.215210] LustreError: 377316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772441777 with bad export cookie 18257179786994325995 [ 6117.217936] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6117.218437] LustreError: 377316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6117.326845] Lustre: server umount lustre-MDT0001 complete [ 6120.417937] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6121.013927] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 03:56:20 (1772441780) [ 6124.108653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6124.322821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6125.788939] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6129.137888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6129.255094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6130.725962] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6132.667308] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6133.957145] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6136.739355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6138.814979] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6140.632127] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6141.255265] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6152.037373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6154.215046] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6154.217091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6154.224549] Lustre: Skipped 1 previous similar message [ 6154.232389] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6159.335458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6159.339054] Lustre: Skipped 2 previous similar messages [ 6159.801587] Lustre: server umount lustre-OST0000 complete [ 6164.449765] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6167.545742] Lustre: server umount lustre-MDT0000 complete [ 6168.730274] LustreError: 381352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772441828 with bad export cookie 18257179786994326870 [ 6168.734177] LustreError: 381352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6168.734305] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6168.851172] Lustre: server umount lustre-MDT0001 complete [ 6172.338642] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 6173.543556] Key type lgssc unregistered [ 6173.698384] LNet: 384758:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6173.700610] LNetError: 384758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6173.713291] LNet: Removed LNI 192.168.201.141@tcp [ 6173.975802] Key type .llcrypt unregistered [ 6173.977025] Key type ._llcrypt unregistered [ 6182.052604] Key type ._llcrypt registered [ 6182.054601] Key type .llcrypt registered [ 6182.088238] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 6182.197318] Key type .llcrypt unregistered [ 6182.199320] Key type ._llcrypt unregistered [ 6189.316460] Key type ._llcrypt registered [ 6189.317619] Key type .llcrypt registered [ 6189.351852] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 6189.665780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6189.703047] alg: No test for adler32 (adler32-zlib) [ 6190.548820] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 6190.634672] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 6192.215236] Key type lgssc registered [ 6192.753503] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6195.926927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6195.932271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6197.089483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6198.431236] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6201.474905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6201.576582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6203.617932] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6205.847990] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6206.944728] LustreError: 386913:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6206.949394] LustreError: 386913:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6207.971085] LustreError: 386931:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6207.975492] LustreError: 386931:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6207.977559] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6208.027391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6208.151144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6209.615779] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6218.208371] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6218.212775] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6218.218347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6218.721664] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6218.723702] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6218.727070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6222.624167] Lustre: server umount lustre-OST0000 complete [ 6223.840447] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6223.840671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6223.841217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6223.848547] Lustre: Skipped 1 previous similar message [ 6228.960042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6228.961737] Lustre: Skipped 2 previous similar messages [ 6229.910375] Lustre: server umount lustre-MDT0000 complete [ 6231.017192] LustreError: 388873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772441891 with bad export cookie 9268213592592250374 [ 6231.020115] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6231.020518] LustreError: 388873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6231.139417] Lustre: server umount lustre-MDT0001 complete [ 6234.200897] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 6235.056131] Key type lgssc unregistered [ 6235.175365] LNet: 389895:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6235.177506] LNetError: 389895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6235.186289] LNet: Removed LNI 192.168.201.141@tcp [ 6235.424793] Key type .llcrypt unregistered [ 6235.425761] Key type ._llcrypt unregistered [ 6243.319856] Key type ._llcrypt registered [ 6243.320797] Key type .llcrypt registered [ 6243.352097] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 6243.634619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6243.687681] alg: No test for adler32 (adler32-zlib) [ 6244.530690] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 6244.612163] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 6246.191201] Key type lgssc registered [ 6246.748100] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6249.488544] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6249.494523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6264.927360] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6270.047210] LustreError: 391245:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.141@tcp: failed processing log, type 4: rc = -110 [ 6296.671207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6299.227917] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6301.727222] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6305.053563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6305.289445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6305.314419] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6306.739611] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6310.371842] LustreError: 391271:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6310.383136] LustreError: 391271:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6310.388361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6310.555136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6312.128334] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6314.418324] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6316.065285] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6320.608405] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6320.613039] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6320.621537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6321.121440] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6321.127625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6321.137896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6322.148999] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6322.152543] Lustre: Skipped 1 previous similar message [ 6324.457635] Lustre: server umount lustre-OST0000 complete [ 6326.240786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6326.243275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6326.246934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6332.384929] LustreError: 392172:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6332.392668] LustreError: 392172:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6332.407173] Lustre: server umount lustre-MDT0000 complete [ 6334.046518] LustreError: 391253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772441994 with bad export cookie 3273606890168637465 [ 6334.052553] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6334.053600] LustreError: 391253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6334.193126] Lustre: server umount lustre-MDT0001 complete [ 6337.994063] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 6339.280445] Key type lgssc unregistered [ 6339.465555] LNet: 394462:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6339.470522] LNetError: 394462:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6339.482556] LNet: Removed LNI 192.168.201.141@tcp [ 6339.932334] Key type .llcrypt unregistered [ 6339.936048] Key type ._llcrypt unregistered [ 6350.340422] Key type ._llcrypt registered [ 6350.341417] Key type .llcrypt registered [ 6350.382138] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 6350.706270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6350.746775] alg: No test for adler32 (adler32-zlib) [ 6351.594037] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 6351.676759] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 6353.255148] Key type lgssc registered [ 6353.765150] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6356.671528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6356.683043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6357.883156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6359.427234] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6362.962174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6363.114180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6364.667451] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6367.227396] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6368.660615] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6370.786446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6370.893654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6372.545558] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6374.243843] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6382.182699] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6387.684957] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6387.686671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6387.693711] Lustre: Skipped 2 previous similar messages [ 6387.699969] Lustre: Skipped 2 previous similar messages [ 6390.059558] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.41@tcp (stopping) [ 6391.918612] Lustre: server umount lustre-MDT0000 complete [ 6392.803116] LustreError: 395838:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6392.813796] LustreError: 395838:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6393.061165] LustreError: 395820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.41@tcp arrived at 1772442053 with bad export cookie 4838428474425597716 [ 6393.069793] LustreError: 395820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6394.006100] LustreError: 395818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772442054 with bad export cookie 4838428474425597065 [ 6394.012891] LustreError: 395818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6394.013136] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6397.920794] LustreError: 396549:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6397.921778] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6397.933708] LustreError: 396549:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6397.939831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6399.967975] LustreError: 395839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6399.968462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6399.972725] LustreError: 395839:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6400.162607] Lustre: server umount lustre-OST0000 complete [ 6403.101251] Lustre: server umount lustre-MDT0001 complete [ 6407.147517] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 6408.399648] Key type lgssc unregistered [ 6408.518678] LNet: 399202:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6408.522589] LNetError: 399202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6408.533528] LNet: Removed LNI 192.168.201.141@tcp [ 6408.840827] Key type .llcrypt unregistered [ 6408.842076] Key type ._llcrypt unregistered [ 6419.049147] Key type ._llcrypt registered [ 6419.050157] Key type .llcrypt registered [ 6419.082227] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 6419.404944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6419.457721] alg: No test for adler32 (adler32-zlib) [ 6420.300140] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 6420.376937] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 6421.951210] Key type lgssc registered [ 6422.362846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6425.748864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6425.760072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6426.946549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6428.447824] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6431.564948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6431.744099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6433.287209] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6435.706879] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6437.268422] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6440.112501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6440.223460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6442.586078] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6444.701981] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6452.519536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6457.312689] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6457.318491] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6457.326679] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6457.827386] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6457.835431] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6460.199780] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.41@tcp (stopping) [ 6462.314518] Lustre: server umount lustre-MDT0001 complete [ 6462.946386] LustreError: 401301:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6462.957074] LustreError: 401301:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6468.064278] LustreError: 400577:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6468.064972] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6468.065581] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6468.066290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6470.311329] Lustre: server umount lustre-OST0000 complete [ 6472.032701] Lustre: server umount lustre-MDT0000 complete [ 6476.981547] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 6478.035225] Key type lgssc unregistered [ 6478.191482] LNet: 403930:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6478.195619] LNetError: 403930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6478.208295] LNet: Removed LNI 192.168.201.141@tcp [ 6478.550545] Key type .llcrypt unregistered [ 6478.552584] Key type ._llcrypt unregistered [ 6488.398419] Key type ._llcrypt registered [ 6488.399718] Key type .llcrypt registered [ 6488.451711] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 6495.085626] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 6495.515919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6495.524313] alg: No test for adler32 (adler32-zlib) [ 6496.388676] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 6496.469921] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 6498.047201] Key type lgssc registered [ 6498.644341] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6514.311537] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 6520.766851] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6520.774556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6521.892048] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6521.912896] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6521.970050] Lustre: lustre-MDT0000: new disk, initializing [ 6522.002435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6522.012424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6523.719284] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6531.278287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6531.409249] Lustre: lustre-OST0000: new disk, initializing [ 6531.411424] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6531.413413] Lustre: Skipped 1 previous similar message [ 6531.437656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6531.600692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6531.604236] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6531.614147] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6533.260929] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6536.950918] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6542.811996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6542.851898] Lustre: 407865:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6542.876973] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6542.919596] Lustre: lustre-MDT0001: new disk, initializing [ 6542.964750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6542.978961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6542.982317] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6544.445759] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6548.443443] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6548.516421] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6549.939670] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6551.071485] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 1 sec [ 6554.919452] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.41@tcp (stopping) [ 6555.107364] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6555.108584] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6555.120253] Lustre: Skipped 1 previous similar message [ 6560.033678] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.41@tcp (stopping) [ 6560.038878] Lustre: Skipped 2 previous similar messages [ 6565.150838] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.41@tcp (stopping) [ 6565.154198] Lustre: Skipped 2 previous similar messages [ 6567.391110] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6567.449137] Lustre: server umount lustre-MDT0001 complete [ 6575.586145] LustreError: 407876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6575.592345] LustreError: 407876:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6580.705291] LustreError: 407875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6580.705635] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6580.726092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6580.729664] Lustre: Skipped 2 previous similar messages [ 6585.825963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6585.826882] LustreError: 407876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6586.349874] Lustre: server umount lustre-OST0000 complete [ 6602.207139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6602.324641] Lustre: server umount lustre-MDT0000 complete [ 6607.318842] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 6608.341976] Key type lgssc unregistered [ 6608.482362] LNet: 412444:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6608.486456] LNetError: 412444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6608.497449] LNet: Removed LNI 192.168.201.141@tcp [ 6608.879153] Key type .llcrypt unregistered [ 6608.881165] Key type ._llcrypt unregistered [ 6623.400843] Key type ._llcrypt registered [ 6623.401905] Key type .llcrypt registered [ 6623.442539] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 6623.562505] Key type .llcrypt unregistered [ 6623.565079] Key type ._llcrypt unregistered [ 6630.020790] Key type ._llcrypt registered [ 6630.021818] Key type .llcrypt registered [ 6630.056125] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 6630.464668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6630.479776] alg: No test for adler32 (adler32-zlib) [ 6631.334570] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 6631.416495] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 6632.999153] Key type lgssc registered [ 6633.500250] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6642.615757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6642.626887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6643.735196] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6643.749181] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6643.792352] Lustre: lustre-MDT0000: new disk, initializing [ 6643.815359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6643.821991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6645.212925] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6649.772271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6649.823051] Lustre: 415758:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6649.857275] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6649.860073] Lustre: Skipped 1 previous similar message [ 6649.904480] Lustre: lustre-MDT0001: new disk, initializing [ 6649.937733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6649.955305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6649.960077] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6651.371077] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6654.816696] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6656.137808] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6658.821176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6658.970114] Lustre: lustre-OST0000: new disk, initializing [ 6658.974184] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6659.022392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6661.093507] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6664.602025] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6667.275341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6667.281048] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6667.317404] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6675.424467] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6675.429458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6675.437904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6677.474560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6677.476925] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6678.701922] Lustre: server umount lustre-MDT0000 complete [ 6680.197599] LustreError: 415750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772442340 with bad export cookie 10969534861947221000 [ 6680.202650] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6680.204116] LustreError: 415750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6680.346734] Lustre: server umount lustre-MDT0001 complete [ 6691.053635] Lustre: server umount lustre-OST0000 complete [ 6694.973242] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 04:05:54 (1772442354) [ 6698.398650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6698.623820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6700.353404] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6704.097683] LustreError: 419412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6704.107381] LustreError: 419412:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6704.320975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6704.500043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6706.146566] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6708.695398] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6710.466614] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6724.398881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6724.530956] Lustre: Found index 0 for lustre-OST0000, updating log [ 6724.661574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6727.271255] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6729.948975] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6744.114194] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6749.159989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6749.160157] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6749.161578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6749.161590] Lustre: Skipped 2 previous similar messages [ 6749.163153] Lustre: Skipped 2 previous similar messages [ 6753.457900] Lustre: server umount lustre-OST0000 complete [ 6766.550252] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6766.555551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6766.557116] Lustre: Skipped 3 previous similar messages [ 6771.168891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6771.173503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6771.181370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6772.715708] Lustre: server umount lustre-MDT0000 complete [ 6774.199132] LustreError: 419393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772442434 with bad export cookie 10969534861947222379 [ 6774.204199] LustreError: 419393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6774.206400] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6774.368914] Lustre: server umount lustre-MDT0001 complete [ 6778.115106] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 04:07:17 (1772442437) [ 6792.223258] LNet: 1 local NIs in recovery (showing 1): 192.168.201.141@tcp [ 7364.995053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7365.241880] LustreError: 425500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7365.280188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7366.988315] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7370.720377] LustreError: 425501:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7370.944950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7371.056418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7372.529599] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7375.092054] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7376.832233] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7391.116997] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7391.253441] Lustre: Found index 0 for lustre-OST0000, updating log [ 7391.381645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7393.450186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 7393.754657] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7396.471861] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7398.500599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7398.515373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7398.519412] Lustre: Skipped 1 previous similar message [ 7399.591312] Lustre: server umount lustre-OST0000 complete [ 7401.952417] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7401.957469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7401.966096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7405.025845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7405.026246] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7407.522106] Lustre: server umount lustre-MDT0000 complete [ 7409.297171] LustreError: 425480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772443069 with bad export cookie 10969534861947223450 [ 7409.302902] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7409.304621] LustreError: 425480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7409.453133] Lustre: server umount lustre-MDT0001 complete [ 7420.959401] LNet: 1 local NIs in recovery (showing 1): 192.168.201.141@tcp [ 7428.721299] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 7432.887282] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 7450.087727] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 04:18:29 (1772443109) [ 7458.008800] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 7459.435924] Key type lgssc unregistered [ 7459.606634] LNet: 431770:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7459.611858] LNetError: 431770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7522.085571] LNet: Removed LNI 192.168.201.141@tcp [ 7522.558272] Key type .llcrypt unregistered [ 7522.560693] Key type ._llcrypt unregistered [ 7530.842718] Key type ._llcrypt registered [ 7530.844601] Key type .llcrypt registered [ 7530.906940] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 7531.219447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7531.225651] alg: No test for adler32 (adler32-zlib) [ 7532.101301] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 7532.182862] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 7533.767238] Key type lgssc registered [ 7534.157827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7536.190394] LNet: Added LNI 192.168.201.141@tcp100 [8/256/0/180] [ 7536.603718] LNet: Added LNI 192.168.201.141@tcp101 [8/256/0/180] [ 7537.040771] LNet: Added LNI 192.168.201.141@tcp102 [8/256/0/180] [ 7537.467733] LNet: Added LNI 192.168.201.141@tcp103 [8/256/0/180] [ 7537.899707] LNet: Added LNI 192.168.201.141@tcp104 [8/256/0/180] [ 7538.327047] LNet: Added LNI 192.168.201.141@tcp105 [8/256/0/180] [ 7538.753415] LNet: Added LNI 192.168.201.141@tcp106 [8/256/0/180] [ 7539.151093] LNet: Added LNI 192.168.201.141@tcp107 [8/256/0/180] [ 7539.564289] LNet: Added LNI 192.168.201.141@tcp108 [8/256/0/180] [ 7540.012346] LNet: Added LNI 192.168.201.141@tcp109 [8/256/0/180] [ 7540.454606] LNet: Added LNI 192.168.201.141@tcp110 [8/256/0/180] [ 7540.876471] LNet: Added LNI 192.168.201.141@tcp111 [8/256/0/180] [ 7541.306479] LNet: Added LNI 192.168.201.141@tcp112 [8/256/0/180] [ 7541.727752] LNet: Added LNI 192.168.201.141@tcp113 [8/256/0/180] [ 7542.154441] LNet: Added LNI 192.168.201.141@tcp114 [8/256/0/180] [ 7542.588814] LNet: Added LNI 192.168.201.141@tcp115 [8/256/0/180] [ 7543.015237] LNet: Added LNI 192.168.201.141@tcp116 [8/256/0/180] [ 7543.440512] LNet: Added LNI 192.168.201.141@tcp117 [8/256/0/180] [ 7543.858470] LNet: Added LNI 192.168.201.141@tcp118 [8/256/0/180] [ 7544.249334] LNet: Added LNI 192.168.201.141@tcp119 [8/256/0/180] [ 7544.679329] LNet: Added LNI 192.168.201.141@tcp120 [8/256/0/180] [ 7545.108161] LNet: Added LNI 192.168.201.141@tcp121 [8/256/0/180] [ 7545.541897] LNet: Added LNI 192.168.201.141@tcp122 [8/256/0/180] [ 7545.980410] LNet: Added LNI 192.168.201.141@tcp123 [8/256/0/180] [ 7546.411032] LNet: Added LNI 192.168.201.141@tcp124 [8/256/0/180] [ 7546.849259] LNet: Added LNI 192.168.201.141@tcp125 [8/256/0/180] [ 7547.279406] LNet: Added LNI 192.168.201.141@tcp126 [8/256/0/180] [ 7547.705494] LNet: Added LNI 192.168.201.141@tcp127 [8/256/0/180] [ 7548.140829] LNet: Added LNI 192.168.201.141@tcp128 [8/256/0/180] [ 7548.548450] LNet: Added LNI 192.168.201.141@tcp129 [8/256/0/180] [ 7548.989125] LNet: Added LNI 192.168.201.141@tcp130 [8/256/0/180] [ 7549.354146] LNet: Added LNI 192.168.201.141@tcp131 [8/256/0/180] [ 7549.790977] LNet: Added LNI 192.168.201.141@tcp132 [8/256/0/180] [ 7550.229257] LNet: Added LNI 192.168.201.141@tcp133 [8/256/0/180] [ 7550.660381] LNet: Added LNI 192.168.201.141@tcp134 [8/256/0/180] [ 7551.106330] LNet: Added LNI 192.168.201.141@tcp135 [8/256/0/180] [ 7554.238765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7554.245477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7555.292906] Lustre: 434751:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.141@tcp: can't add NID 192.168.201.141@tcp130: rc = -75 [ 7555.367653] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7555.380711] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7555.426592] Lustre: lustre-MDT0000: new disk, initializing [ 7555.456917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7555.464748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7557.049100] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7562.189271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7562.244986] Lustre: 434768:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7562.272736] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7562.276048] Lustre: Skipped 1 previous similar message [ 7562.316476] Lustre: lustre-MDT0001: new disk, initializing [ 7562.328327] Lustre: 435551:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.141@tcp: can't add NID 192.168.201.141@tcp130: rc = -75 [ 7562.331400] Lustre: 435551:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7562.374260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7562.386369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7562.389779] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7564.067864] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7564.866268] Lustre: 435899:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.141@tcp: can't add NID 192.168.201.141@tcp130: rc = -75 [ 7564.872327] Lustre: 435899:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7567.515985] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7569.051949] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7572.040819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7572.139807] Lustre: lustre-OST0000: new disk, initializing [ 7572.141553] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7572.169069] Lustre: 436520:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.141@tcp: can't add NID 192.168.201.141@tcp130: rc = -75 [ 7572.171622] Lustre: 436520:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 7572.186691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7574.506177] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7577.601498] Lustre: 437064:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.141@tcp: can't add NID 192.168.201.141@tcp130: rc = -75 [ 7577.607834] Lustre: 437064:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7577.631607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7577.639904] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7577.706347] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7578.387880] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7582.690979] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7582.694470] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7582.698508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7582.704998] Lustre: Skipped 1 previous similar message [ 7583.914442] Lustre: server umount lustre-OST0000 complete [ 7587.809907] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7587.811354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7587.818607] Lustre: Skipped 1 previous similar message [ 7587.822578] Lustre: Skipped 1 previous similar message [ 7591.845219] Lustre: server umount lustre-MDT0000 complete [ 7593.648259] LustreError: 434759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772443253 with bad export cookie 6950426820337483599 [ 7593.653229] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7593.655838] LustreError: 434759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7593.818755] Lustre: server umount lustre-MDT0001 complete [ 7598.663251] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 7600.052216] Key type lgssc unregistered [ 7600.273979] LNet: 438295:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7600.279690] LNetError: 438295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7600.284910] LNet: Removed LNI 192.168.201.141@tcp [ 7601.311159] LNet: Removed LNI 192.168.201.141@tcp100 [ 7601.314143] LNetError: 438295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7601.319526] LNetError: 438295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7601.325459] LNet: Removed LNI 192.168.201.141@tcp101 [ 7602.335153] LNet: Removed LNI 192.168.201.141@tcp102 [ 7602.338442] LNetError: 438295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7602.343771] LNetError: 438295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7602.349617] LNet: Removed LNI 192.168.201.141@tcp103 [ 7603.359150] LNet: Removed LNI 192.168.201.141@tcp104 [ 7604.383194] LNet: Removed LNI 192.168.201.141@tcp105 [ 7604.386418] LNetError: 438295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7604.391794] LNetError: 438295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7604.397612] LNet: Removed LNI 192.168.201.141@tcp106 [ 7605.407155] LNet: Removed LNI 192.168.201.141@tcp107 [ 7606.431164] LNet: Removed LNI 192.168.201.141@tcp108 [ 7607.455114] LNet: Removed LNI 192.168.201.141@tcp109 [ 7608.479162] LNet: Removed LNI 192.168.201.141@tcp110 [ 7608.481923] LNetError: 438295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7608.486683] LNetError: 438295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7608.491922] LNet: Removed LNI 192.168.201.141@tcp111 [ 7609.503169] LNet: Removed LNI 192.168.201.141@tcp112 [ 7610.527183] LNet: Removed LNI 192.168.201.141@tcp113 [ 7611.551172] LNet: Removed LNI 192.168.201.141@tcp114 [ 7611.553921] LNet: Removed LNI 192.168.201.141@tcp115 [ 7611.556501] LNet: Removed LNI 192.168.201.141@tcp116 [ 7611.559188] LNet: Removed LNI 192.168.201.141@tcp117 [ 7611.561707] LNet: Removed LNI 192.168.201.141@tcp118 [ 7611.564371] LNet: Removed LNI 192.168.201.141@tcp119 [ 7611.566932] LNet: Removed LNI 192.168.201.141@tcp120 [ 7611.569482] LNet: Removed LNI 192.168.201.141@tcp121 [ 7611.572121] LNet: Removed LNI 192.168.201.141@tcp122 [ 7611.574649] LNet: Removed LNI 192.168.201.141@tcp123 [ 7611.577371] LNet: Removed LNI 192.168.201.141@tcp124 [ 7611.579911] LNet: Removed LNI 192.168.201.141@tcp125 [ 7611.582440] LNet: Removed LNI 192.168.201.141@tcp126 [ 7611.585126] LNet: Removed LNI 192.168.201.141@tcp127 [ 7611.587648] LNet: Removed LNI 192.168.201.141@tcp128 [ 7611.590331] LNet: Removed LNI 192.168.201.141@tcp129 [ 7611.592916] LNet: Removed LNI 192.168.201.141@tcp130 [ 7611.595406] LNet: Removed LNI 192.168.201.141@tcp131 [ 7611.597941] LNet: Removed LNI 192.168.201.141@tcp132 [ 7611.600481] LNet: Removed LNI 192.168.201.141@tcp133 [ 7611.603087] LNet: Removed LNI 192.168.201.141@tcp134 [ 7611.611833] LNet: Removed LNI 192.168.201.141@tcp135 [ 7612.013674] Key type .llcrypt unregistered [ 7612.016086] Key type ._llcrypt unregistered [ 7622.264688] Key type ._llcrypt registered [ 7622.266257] Key type .llcrypt registered [ 7622.322107] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 7630.388821] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 7630.779995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7630.788744] alg: No test for adler32 (adler32-zlib) [ 7631.636558] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 7631.712860] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 7633.295157] Key type lgssc registered [ 7633.865213] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7650.791892] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 04:21:50 (1772443310) [ 7658.844319] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 7660.311651] Key type lgssc unregistered [ 7660.467471] LNet: 442302:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7660.472316] LNetError: 442302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7660.480484] LNet: Removed LNI 192.168.201.141@tcp [ 7660.921756] Key type .llcrypt unregistered [ 7660.924052] Key type ._llcrypt unregistered [ 7669.284026] Key type ._llcrypt registered [ 7669.285608] Key type .llcrypt registered [ 7669.345727] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 7669.715629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7669.738446] alg: No test for adler32 (adler32-zlib) [ 7670.603151] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 7670.679826] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 7672.263151] Key type lgssc registered [ 7672.818508] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7677.363855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7677.374152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7678.506809] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7678.525362] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7678.574733] Lustre: lustre-MDT0000: new disk, initializing [ 7678.614648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7678.627340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7680.394148] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7685.987124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7686.064104] Lustre: 443476:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7686.111761] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7686.115331] Lustre: Skipped 1 previous similar message [ 7686.164888] Lustre: lustre-MDT0001: new disk, initializing [ 7686.222374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7686.245767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7686.252784] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7688.077258] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7692.205848] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7694.070512] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7697.324865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7697.473813] Lustre: lustre-OST0000: new disk, initializing [ 7697.477877] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7697.525894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7700.153429] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7702.530677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7702.535231] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7702.574728] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7703.982304] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7705.506363] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing lnet_if_list [ 7707.617401] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7707.636529] LNet: Added LNI 192.168.201.141@tcp10 [8/256/0/180] [ 7709.805568] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7709.820040] LNet: Added LNI 192.168.201.141@tcp11 [8/256/0/180] [ 7711.464803] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7711.484358] LNet: Added LNI 192.168.201.141@tcp12 [8/256/0/180] [ 7712.949366] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7712.959557] LNet: Added LNI 192.168.201.141@tcp13 [8/256/0/180] [ 7714.579684] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7714.598891] LNet: Added LNI 192.168.201.141@tcp14 [8/256/0/180] [ 7716.383843] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7716.398814] LNet: Added LNI 192.168.201.141@tcp15 [8/256/0/180] [ 7718.243545] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7718.262671] LNet: Added LNI 192.168.201.141@tcp16 [8/256/0/180] [ 7719.538772] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7719.552976] LNet: Added LNI 192.168.201.141@tcp17 [8/256/0/180] [ 7721.506878] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7721.522767] LNet: Added LNI 192.168.201.141@tcp18 [8/256/0/180] [ 7723.532195] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7723.545781] LNet: Added LNI 192.168.201.141@tcp19 [8/256/0/180] [ 7725.543405] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7725.557622] LNet: Added LNI 192.168.201.141@tcp20 [8/256/0/180] [ 7727.647926] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7727.670262] LNet: Added LNI 192.168.201.141@tcp21 [8/256/0/180] [ 7729.531054] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7729.546782] LNet: Added LNI 192.168.201.141@tcp22 [8/256/0/180] [ 7731.774366] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7731.790141] LNet: Added LNI 192.168.201.141@tcp23 [8/256/0/180] [ 7734.036656] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7734.054993] LNet: Added LNI 192.168.201.141@tcp24 [8/256/0/180] [ 7736.294457] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7736.309882] LNet: Added LNI 192.168.201.141@tcp25 [8/256/0/180] [ 7738.766736] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7738.785626] LNet: Added LNI 192.168.201.141@tcp26 [8/256/0/180] [ 7741.087424] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7741.106860] LNet: Added LNI 192.168.201.141@tcp27 [8/256/0/180] [ 7743.472145] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7743.484427] LNet: Added LNI 192.168.201.141@tcp28 [8/256/0/180] [ 7745.635344] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7745.654913] LNet: Added LNI 192.168.201.141@tcp29 [8/256/0/180] [ 7748.355388] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7748.373511] LNet: Added LNI 192.168.201.141@tcp30 [8/256/0/180] [ 7750.838573] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7750.856452] LNet: Added LNI 192.168.201.141@tcp31 [8/256/0/180] [ 7753.510251] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7753.529703] LNet: Added LNI 192.168.201.141@tcp32 [8/256/0/180] [ 7755.803971] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7755.818128] LNet: Added LNI 192.168.201.141@tcp33 [8/256/0/180] [ 7758.526939] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7758.546950] LNet: Added LNI 192.168.201.141@tcp34 [8/256/0/180] [ 7761.620478] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7761.635312] LNet: Added LNI 192.168.201.141@tcp35 [8/256/0/180] [ 7764.323342] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7764.339854] LNet: Added LNI 192.168.201.141@tcp36 [8/256/0/180] [ 7767.245505] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7767.261321] LNet: Added LNI 192.168.201.141@tcp37 [8/256/0/180] [ 7770.166261] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7770.184583] LNet: Added LNI 192.168.201.141@tcp38 [8/256/0/180] [ 7772.865162] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7772.882364] LNet: Added LNI 192.168.201.141@tcp39 [8/256/0/180] [ 7775.708320] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7775.726612] LNet: Added LNI 192.168.201.141@tcp40 [8/256/0/180] [ 7778.743374] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7778.760421] LNet: Added LNI 192.168.201.141@tcp41 [8/256/0/180] [ 7781.852710] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7781.870463] LNet: Added LNI 192.168.201.141@tcp42 [8/256/0/180] [ 7784.971435] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7784.990453] LNet: Added LNI 192.168.201.141@tcp43 [8/256/0/180] [ 7788.195239] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7788.208869] LNet: Added LNI 192.168.201.141@tcp44 [8/256/0/180] [ 7791.492135] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7791.507311] LNet: Added LNI 192.168.201.141@tcp45 [8/256/0/180] [ 7794.307315] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7794.318218] LNet: Added LNI 192.168.201.141@tcp46 [8/256/0/180] [ 7796.881567] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7796.896509] LNet: Added LNI 192.168.201.141@tcp47 [8/256/0/180] [ 7799.627371] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7799.648605] LNet: Added LNI 192.168.201.141@tcp48 [8/256/0/180] [ 7802.861290] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7802.875974] LNet: Added LNI 192.168.201.141@tcp49 [8/256/0/180] [ 7806.036529] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7806.050774] LNet: Added LNI 192.168.201.141@tcp50 [8/256/0/180] [ 7817.970882] Lustre: server umount lustre-MDT0000 complete [ 7819.232434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7819.237296] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7819.651845] LustreError: 444255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772443479 with bad export cookie 12060713560718628516 [ 7819.657611] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7819.658949] LustreError: 444255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7819.816868] Lustre: server umount lustre-MDT0001 complete [ 7829.142319] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 7832.841889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7836.127200] Lustre: 442747:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772443480/real 1772443480] req@000000004adc120a x1858541537288448/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772443496 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7836.127328] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7836.142030] Lustre: 442747:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7836.153872] Lustre: Skipped 1 previous similar message [ 7840.223171] Lustre: 442750:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772443484/real 1772443484] req@00000000451943d0 x1858541537288704/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772443500 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7840.237706] Lustre: 442750:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7844.383545] LustreError: 442746:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000451943d0 x1858541537289728/t0(0) o250->MGC192.168.201.141@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7844.541151] LustreError: 445213:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7844.554649] LustreError: 445213:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7844.564488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7845.599120] Lustre: 442747:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772443489/real 1772443489] req@000000004b2517ba x1858541537288960/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772443505 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7845.606582] Lustre: 442747:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7846.160083] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7846.628298] LustreError: 445214:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7846.630684] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7846.638231] LustreError: 445214:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7850.271885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7850.441557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7851.423144] Lustre: 442750:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772443494/real 1772443494] req@00000000137007e6 x1858541537289344/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772443510 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7852.171513] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7854.830667] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7855.590382] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7856.503658] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7868.140595] Lustre: server umount lustre-OST0000 complete [ 7870.946875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7870.947309] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7870.948213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7870.954224] Lustre: Skipped 1 previous similar message [ 7876.066275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7876.070058] Lustre: Skipped 2 previous similar messages [ 7881.185428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7881.189734] Lustre: Skipped 1 previous similar message [ 7884.255190] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7884.341652] Lustre: server umount lustre-MDT0000 complete [ 7886.071104] LustreError: 443469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772443546 with bad export cookie 12060713560718630651 [ 7886.077965] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7886.079214] LustreError: 443469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7896.624779] Lustre: server umount lustre-MDT0001 complete [ 7901.449553] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 7902.901395] Key type lgssc unregistered [ 7903.101823] LNet: 471164:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7903.108356] LNetError: 471164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7903.112906] LNet: Removed LNI 192.168.201.141@tcp [ 7903.115329] LNet: Removed LNI 192.168.201.141@tcp10 [ 7903.117785] LNet: Removed LNI 192.168.201.141@tcp11 [ 7903.120222] LNet: Removed LNI 192.168.201.141@tcp12 [ 7903.122659] LNet: Removed LNI 192.168.201.141@tcp13 [ 7903.125156] LNet: Removed LNI 192.168.201.141@tcp14 [ 7903.127636] LNet: Removed LNI 192.168.201.141@tcp15 [ 7903.130211] LNet: Removed LNI 192.168.201.141@tcp16 [ 7903.133198] LNet: Removed LNI 192.168.201.141@tcp17 [ 7903.135780] LNet: Removed LNI 192.168.201.141@tcp18 [ 7903.138152] LNet: Removed LNI 192.168.201.141@tcp19 [ 7903.140583] LNet: Removed LNI 192.168.201.141@tcp20 [ 7903.143133] LNet: Removed LNI 192.168.201.141@tcp21 [ 7903.145541] LNet: Removed LNI 192.168.201.141@tcp22 [ 7903.148055] LNet: Removed LNI 192.168.201.141@tcp23 [ 7903.150459] LNet: Removed LNI 192.168.201.141@tcp24 [ 7903.152956] LNet: Removed LNI 192.168.201.141@tcp25 [ 7903.155353] LNet: Removed LNI 192.168.201.141@tcp26 [ 7903.157670] LNet: Removed LNI 192.168.201.141@tcp27 [ 7903.161327] LNet: Removed LNI 192.168.201.141@tcp28 [ 7903.163560] LNet: Removed LNI 192.168.201.141@tcp29 [ 7903.165840] LNet: Removed LNI 192.168.201.141@tcp30 [ 7903.168048] LNet: Removed LNI 192.168.201.141@tcp31 [ 7903.170168] LNet: Removed LNI 192.168.201.141@tcp32 [ 7903.172334] LNet: Removed LNI 192.168.201.141@tcp33 [ 7903.174487] LNet: Removed LNI 192.168.201.141@tcp34 [ 7903.176552] LNet: Removed LNI 192.168.201.141@tcp35 [ 7903.178583] LNet: Removed LNI 192.168.201.141@tcp36 [ 7903.180612] LNet: Removed LNI 192.168.201.141@tcp37 [ 7903.182617] LNet: Removed LNI 192.168.201.141@tcp38 [ 7903.184712] LNet: Removed LNI 192.168.201.141@tcp39 [ 7903.186552] LNet: Removed LNI 192.168.201.141@tcp40 [ 7903.188466] LNet: Removed LNI 192.168.201.141@tcp41 [ 7903.190292] LNet: Removed LNI 192.168.201.141@tcp42 [ 7903.192097] LNet: Removed LNI 192.168.201.141@tcp43 [ 7903.193817] LNet: Removed LNI 192.168.201.141@tcp44 [ 7903.195429] LNet: Removed LNI 192.168.201.141@tcp45 [ 7903.197103] LNet: Removed LNI 192.168.201.141@tcp46 [ 7903.198758] LNet: Removed LNI 192.168.201.141@tcp47 [ 7903.200446] LNet: Removed LNI 192.168.201.141@tcp48 [ 7903.202149] LNet: Removed LNI 192.168.201.141@tcp49 [ 7903.209389] LNet: Removed LNI 192.168.201.141@tcp50 [ 7903.608929] Key type .llcrypt unregistered [ 7903.611255] Key type ._llcrypt unregistered [ 7913.267936] Key type ._llcrypt registered [ 7913.269367] Key type .llcrypt registered [ 7913.305821] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 7913.712336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7913.717733] alg: No test for adler32 (adler32-zlib) [ 7914.575807] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 7914.656021] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 7916.239138] Key type lgssc registered [ 7916.849833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7920.559700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7920.570500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7921.788722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7923.520142] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7927.521839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7927.668912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7929.283834] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7931.614550] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7933.271503] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7936.539321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7936.680763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7939.186832] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7941.762829] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7946.926128] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7951.606688] loop0: detected capacity change from 0 to 204800000 [ 7951.696817] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7951.701446] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7951.706719] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7954.202312] loop0: detected capacity change from 0 to 204800000 [ 7954.256064] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7954.260058] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7954.263943] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7956.771958] loop0: detected capacity change from 0 to 204800000 [ 7959.399680] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7959.492152] Lustre: fs15246-OST0000: new disk, initializing [ 7959.496161] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7959.527893] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7962.114814] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7966.150921] loop1: detected capacity change from 0 to 204800000 [ 7968.250298] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7968.296640] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7968.324070] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7968.368141] Lustre: fs15246-MDT0000: new disk, initializing [ 7968.422930] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7968.433075] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7968.477904] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7969.504174] Lustre: 476748:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000bec2b068 x1858541793083904/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1772443645 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7969.947485] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7974.569173] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7974.581731] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7974.599564] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7984.610730] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7984.626888] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7989.735975] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7990.751443] Lustre: server umount fs15246-MDT0000 complete [ 7994.851446] LustreError: 472411:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7994.867523] LustreError: 472411:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7997.215310] Lustre: server umount fs15246-OST0000 complete [ 8003.552093] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8003.554432] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8003.558717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8004.576390] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8004.579578] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8010.209131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8010.211547] Lustre: Skipped 4 previous similar messages [ 8015.329430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8015.331657] Lustre: Skipped 1 previous similar message [ 8016.863129] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8016.916164] Lustre: server umount lustre-OST0000 complete [ 8019.935643] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8019.937642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8024.217974] Lustre: server umount lustre-MDT0000 complete [ 8025.367139] LustreError: 472366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772443685 with bad export cookie 7199050706979509837 [ 8025.369984] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8025.370447] LustreError: 472366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8025.483712] Lustre: server umount lustre-MDT0001 complete [ 8028.972512] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 8029.991026] Key type lgssc unregistered [ 8030.116336] LNet: 479809:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8030.118492] LNetError: 479809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8030.127312] LNet: Removed LNI 192.168.201.141@tcp [ 8030.422453] Key type .llcrypt unregistered [ 8030.424179] Key type ._llcrypt unregistered [ 8037.802734] Key type ._llcrypt registered [ 8037.803895] Key type .llcrypt registered [ 8037.849403] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 8038.298680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8038.306548] alg: No test for adler32 (adler32-zlib) [ 8039.179184] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 8039.263883] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 8040.847197] Key type lgssc registered [ 8041.249763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8054.427667] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 8057.457466] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 8068.857926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8068.862817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8069.947848] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8069.959960] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8069.997968] Lustre: lustre-MDT0000: new disk, initializing [ 8070.022138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8070.028844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8071.276432] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 8075.247057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8075.285829] Lustre: 484192:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8075.306606] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8075.308647] Lustre: Skipped 1 previous similar message [ 8075.346857] Lustre: lustre-MDT0001: new disk, initializing [ 8075.375610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8075.387248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8075.390651] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8076.541738] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 8079.581985] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8080.725800] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8082.957223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8083.084646] Lustre: lustre-OST0000: new disk, initializing [ 8083.087142] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8083.120290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8085.218633] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 8088.391712] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8089.851554] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8092.664651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8092.668477] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8092.699354] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8092.977885] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 8094.313299] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8094.391873] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8097.762584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8097.763264] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8097.763617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8097.766365] Lustre: Skipped 1 previous similar message [ 8101.201666] Lustre: server umount lustre-OST0000 complete [ 8102.880189] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8102.880827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8102.883756] Lustre: Skipped 1 previous similar message [ 8102.885573] Lustre: Skipped 1 previous similar message [ 8108.000749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8108.003726] Lustre: Skipped 1 previous similar message [ 8108.636617] Lustre: server umount lustre-MDT0000 complete [ 8109.976259] LustreError: 484184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772443770 with bad export cookie 14518409406152410767 [ 8109.979516] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8109.981039] LustreError: 484184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8110.100570] Lustre: server umount lustre-MDT0001 complete [ 8118.440408] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 8121.692752] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 8132.181517] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7851 sec ========== 04:29:51 (1772443791) [ 8132.865547] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:29:52 (1772443792) === [ 8134.074827] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:29:53 (1772443793) === [ 8147.025287] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 8148.086896] Key type lgssc unregistered [ 8148.228559] LNet: 492386:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8148.232343] LNetError: 492386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8148.246638] LNet: Removed LNI 192.168.201.141@tcp [ 8148.568214] Key type .llcrypt unregistered [ 8148.569654] Key type ._llcrypt unregistered