[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 801617501 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001028] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002000] Switched APIC routing to physical x2apic. [ 0.002000] kvm-guest: setup PV IPIs [ 0.002000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.002000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.002000] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.002000] pid_max: default: 32768 minimum: 301 [ 0.002000] LSM: Security Framework initializing [ 0.002000] Yama: becoming mindful. [ 0.002000] SELinux: Initializing. [ 0.002000] *** VALIDATE selinux *** [ 0.002000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.002000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.002000] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.002000] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.002000] *** VALIDATE tmpfs *** [ 0.002000] *** VALIDATE proc *** [ 0.002000] *** VALIDATE cgroup *** [ 0.002000] *** VALIDATE cgroup2 *** [ 0.002000] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.002000] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.002000] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.002000] Spectre V2 : User space: Vulnerable [ 0.002000] Speculative Store Bypass: Vulnerable [ 0.002000] debug: unmapping init [mem 0xffffffff92259000-0xffffffff92260fff] [ 0.002000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.002791] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.003026] ... version: 2 [ 0.004014] ... bit width: 48 [ 0.005009] ... generic registers: 4 [ 0.006011] ... value mask: 0000ffffffffffff [ 0.007019] ... max period: 00007fffffffffff [ 0.008018] ... fixed-purpose events: 3 [ 0.009014] ... event mask: 000000070000000f [ 0.010435] rcu: Hierarchical SRCU implementation. [ 0.012998] smp: Bringing up secondary CPUs ... [ 0.014600] x86: Booting SMP configuration: [ 0.015060] .... node #0, CPUs: #1 #2 #3 [ 0.024210] smp: Brought up 1 node, 4 CPUs [ 0.026020] smpboot: Max logical packages: 1 [ 0.027032] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.063832] node 0 deferred pages initialised in 34ms [ 0.067153] devtmpfs: initialized [ 0.068270] x86/mm: Memory block size: 128MB [ 0.071069] gcov: version magic: 0x41383552 [ 0.073646] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.074082] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.075346] pinctrl core: initialized pinctrl subsystem [ 0.076210] [ 0.076842] ************************************************************* [ 0.077013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.078009] ** ** [ 0.079008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.080011] ** ** [ 0.081016] ** This means that this kernel is built to expose internal ** [ 0.082010] ** IOMMU data structures, which may compromise security on ** [ 0.083010] ** your system. ** [ 0.084015] ** ** [ 0.085020] ** If you see this message and you are not debugging the ** [ 0.086012] ** kernel, report this immediately to your vendor! ** [ 0.087009] ** ** [ 0.088009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.089007] ************************************************************* [ 0.090723] NET: Registered protocol family 16 [ 0.091487] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.092054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.093081] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.094535] cpuidle: using governor menu [ 0.097352] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.098429] PCI: Using configuration type 1 for base access [ 0.099129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.106113] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.107017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.110066] cryptd: max_cpu_qlen set to 1000 [ 0.113021] ACPI: Added _OSI(Module Device) [ 0.113021] ACPI: Added _OSI(Processor Device) [ 0.114007] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.116008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.120575] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.126746] ACPI: Interpreter enabled [ 0.127077] ACPI: PM: (supports S0 S3 S4 S5) [ 0.128007] ACPI: Using IOAPIC for interrupt routing [ 0.129193] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.131000] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.140427] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.141026] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.142013] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.143069] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.145469] acpiphp: Slot [2] registered [ 0.146147] acpiphp: Slot [5] registered [ 0.147092] acpiphp: Slot [6] registered [ 0.148082] acpiphp: Slot [7] registered [ 0.149105] acpiphp: Slot [8] registered [ 0.150077] acpiphp: Slot [9] registered [ 0.151161] acpiphp: Slot [10] registered [ 0.152126] acpiphp: Slot [3] registered [ 0.153067] acpiphp: Slot [4] registered [ 0.154190] acpiphp: Slot [11] registered [ 0.155081] acpiphp: Slot [12] registered [ 0.156056] acpiphp: Slot [13] registered [ 0.157151] acpiphp: Slot [14] registered [ 0.158083] acpiphp: Slot [15] registered [ 0.159130] acpiphp: Slot [16] registered [ 0.160068] acpiphp: Slot [17] registered [ 0.161145] acpiphp: Slot [18] registered [ 0.162172] acpiphp: Slot [19] registered [ 0.163169] acpiphp: Slot [20] registered [ 0.164094] acpiphp: Slot [21] registered [ 0.165066] acpiphp: Slot [22] registered [ 0.166100] acpiphp: Slot [23] registered [ 0.167139] acpiphp: Slot [24] registered [ 0.168096] acpiphp: Slot [25] registered [ 0.169081] acpiphp: Slot [26] registered [ 0.170084] acpiphp: Slot [27] registered [ 0.171142] acpiphp: Slot [28] registered [ 0.172053] acpiphp: Slot [29] registered [ 0.173061] acpiphp: Slot [30] registered [ 0.174057] acpiphp: Slot [31] registered [ 0.175041] PCI host bridge to bus 0000:00 [ 0.176067] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.177073] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.178016] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.179065] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.180010] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.181013] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.182158] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.183864] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.185262] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.191011] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.194070] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.195011] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.196029] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.197018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.198455] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.199000] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.204044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.207095] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.213014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.225012] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.230013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.235537] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.242015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.247012] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.318017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.330000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.332000] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.332016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.343015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.360228] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.373020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.387022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.426022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.458000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.464020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.469020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.488000] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.499054] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.509017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.519017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.567016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.585019] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.599023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.616018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.650015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.670289] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.673515] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.674000] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.674414] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.676192] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.681047] iommu: Default domain type: Passthrough [ 0.682277] SCSI subsystem initialized [ 0.683175] ACPI: bus type USB registered [ 0.685108] usbcore: registered new interface driver usbfs [ 0.686074] usbcore: registered new interface driver hub [ 0.687156] usbcore: registered new device driver usb [ 0.689169] pps_core: LinuxPPS API ver. 1 registered [ 0.691013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.694058] PTP clock support registered [ 0.696199] EDAC MC: Ver: 3.0.0 [ 0.697161] PCI: Using ACPI for IRQ routing [ 0.698913] NetLabel: Initializing [ 0.701008] NetLabel: domain hash size = 128 [ 0.702007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.704074] NetLabel: unlabeled traffic allowed by default [ 0.706281] vgaarb: loaded [ 0.708032] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.709008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.716905] clocksource: Switched to clocksource kvm-clock [ 0.874231] VFS: Disk quotas dquot_6.6.0 [ 0.875453] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.877613] *** VALIDATE ramfs *** [ 0.878893] *** VALIDATE hugetlbfs *** [ 0.880394] pnp: PnP ACPI init [ 0.882525] pnp: PnP ACPI: found 6 devices [ 0.897296] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.900211] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.903845] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.905518] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.907387] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.909241] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.911521] NET: Registered protocol family 2 [ 0.914328] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.932133] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.944304] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.959552] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.964315] TCP: Hash tables configured (established 65536 bind 65536) [ 0.968130] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.971487] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.975376] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.978707] NET: Registered protocol family 1 [ 0.983284] RPC: Registered named UNIX socket transport module. [ 0.986147] RPC: Registered udp transport module. [ 0.988477] RPC: Registered tcp transport module. [ 0.991797] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.994303] NET: Registered protocol family 44 [ 0.995839] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.997426] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.999309] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.001931] PCI: CLS 0 bytes, default 64 [ 1.004291] Unpacking initramfs... [ 4.811401] debug: unmapping init [mem 0xffff9945fcc54000-0xffff9945fffbffff] [ 4.821071] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.827452] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.863020] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 6.877489] Initialise system trusted keyrings [ 6.881508] Key type blacklist registered [ 7.008272] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 7.034270] zbud: loaded [ 7.044989] *** VALIDATE nfs *** [ 7.049037] *** VALIDATE nfs4 *** [ 7.052482] pstore: using deflate compression [ 7.059831] Platform Keyring initialized [ 7.316068] NET: Registered protocol family 38 [ 7.317793] Key type asymmetric registered [ 7.319366] Asymmetric key parser 'x509' registered [ 7.321301] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 7.324159] io scheduler mq-deadline registered [ 7.325876] io scheduler kyber registered [ 7.327466] io scheduler bfq registered [ 7.664758] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 7.668263] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 7.670985] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 7.673789] ACPI: Power Button [PWRF] [ 7.679380] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 7.688277] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 7.737188] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 7.753324] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 7.830737] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 7.965837] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 8.061139] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 8.068588] Non-volatile memory driver v1.3 [ 8.071339] Linux agpgart interface v0.103 [ 8.211084] virtio_blk virtio1: [vda] 134144 512-byte logical blocks (68.7 MB/65.5 MiB) [ 8.220934] vda: detected capacity change from 0 to 68681728 [ 8.259955] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 8.266890] vdb: detected capacity change from 0 to 1073741824 [ 8.313088] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.322653] vdc: detected capacity change from 0 to 2621440000 [ 8.380693] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.383288] vdd: detected capacity change from 0 to 2621440000 [ 8.453416] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.462621] vde: detected capacity change from 0 to 4294967296 [ 8.524314] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.527400] vdf: detected capacity change from 0 to 4294967296 [ 8.553843] libphy: Fixed MDIO Bus: probed [ 8.575867] usbcore: registered new interface driver usbserial_generic [ 8.581630] usbserial: USB Serial support registered for generic [ 8.590093] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 8.606274] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 8.608381] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 8.622420] mousedev: PS/2 mouse device common for all mice [ 8.635705] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 8.641944] rtc_cmos 00:05: RTC can wake from S4 [ 8.651456] rtc_cmos 00:05: registered as rtc0 [ 8.656151] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 8.671817] intel_pstate: CPU model not supported [ 8.679972] hid: raw HID events driver (C) Jiri Kosina [ 8.685524] usbcore: registered new interface driver usbhid [ 8.694735] usbhid: USB HID core driver [ 8.698236] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 8.706358] drop_monitor: Initializing network drop monitor service [ 8.717195] Initializing XFRM netlink socket [ 8.727470] NET: Registered protocol family 10 [ 8.742525] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 8.752399] Segment Routing with IPv6 [ 8.759702] NET: Registered protocol family 17 [ 8.763977] mpls_gso: MPLS GSO support [ 8.782078] RAS: Correctable Errors collector initialized. [ 8.787709] AVX version of gcm_enc/dec engaged. [ 8.791692] AES CTR mode by8 optimization enabled [ 9.202169] sched_clock: Marking stable (9202074302, 0)->(11030327269, -1828252967) [ 9.224412] registered taskstats version 1 [ 9.227074] Loading compiled-in X.509 certificates [ 9.229431] zswap: loaded using pool lzo/zbud [ 9.339679] Key type big_key registered [ 9.393222] Key type encrypted registered [ 9.400130] ima: No TPM chip found, activating TPM-bypass! [ 9.408720] ima: Allocated hash algorithm: sha1 [ 9.410053] ima: No architecture policies found [ 9.412022] evm: Initialising EVM extended attributes: [ 9.420214] evm: security.selinux [ 9.425170] evm: security.ima [ 9.431700] evm: security.capability [ 9.436661] evm: HMAC attrs: 0x1 [ 9.445473] rtc_cmos 00:05: setting system clock to 2026-05-09 17:14:48 UTC (1778346888) [ 9.469945] debug: unmapping init [mem 0xffffffff93203000-0xffffffff933fffff] [ 9.489723] debug: unmapping init [mem 0xffffffff91f82000-0xffffffff92258fff] [ 9.513733] Write protecting the kernel read-only data: 28672k [ 9.523635] debug: unmapping init [mem 0xffffffff90603000-0xffffffff907fffff] [ 9.540777] debug: unmapping init [mem 0xffffffff90f14000-0xffffffff90ffffff] [ 9.729371] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 9.764161] systemd[1]: Detected virtualization kvm. [ 9.765718] systemd[1]: Detected architecture x86-64. [ 9.767296] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 9.839905] systemd[1]: No hostname configured. [ 9.843960] systemd[1]: Set hostname to . [ 9.848472] random: systemd: uninitialized urandom read (16 bytes read) [ 9.857866] systemd[1]: Initializing machine ID from random generator. [ 10.008339] random: ln: uninitialized urandom read (6 bytes read) [ 10.465360] random: systemd: uninitialized urandom read (16 bytes read) [ 10.497469] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 10.527727] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 10.554731] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 13.060910] device-mapper: uevent: version 1.0.3 [ 13.083851] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. [ 15.001394] random: fast init done Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 16.312652] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 17.104307] scsi host0: ata_piix [ 17.166154] scsi host1: ata_piix [ 17.170888] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 17.181135] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 22.011200] random: crng init done [ 22.012570] random: 7 urandom warning(s) missed due to ratelimiting [ 25.999577] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 28.280809] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 32.285791] printk: systemd: 26 output lines suppressed due to ratelimiting [ 33.627836] SELinux: Disabled at runtime. [ 33.777601] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 33.805465] systemd[1]: Detected virtualization kvm. [ 33.810732] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 36.060819] systemd[1]: initrd-switch-root.service: Succeeded. [ 36.075940] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 36.092959] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 36.109200] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 36.111611] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 36.168624] systemd[1]: Starting Journal Service... Starting Journal Service... [ 36.183914] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 36.612085] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 38.291682] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 40.070666] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 40.317052] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 41.196343] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 41.475731] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit)[ 49.009494] Key type dns_resolver registered [ **] A start job is running for Configur…only root support (13s / no limit) [ ***] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit)[ 50.849056] NFS: Registering the id_resolver key type [ 50.860061] Key type id_resolver registered [ 50.861272] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (15s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ 56.930532] hrtimer: interrupt took 22536597 ns [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg440-server login: [ 116.367607] libcfs: loading out-of-tree module taints kernel. [ 116.398733] Key type ._llcrypt registered [ 116.400924] Key type .llcrypt registered [ 116.512067] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 130.707510] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 131.965848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 131.974649] alg: No test for adler32 (adler32-zlib) [ 133.235396] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 133.795034] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 135.519111] Key type lgssc registered [ 136.912042] Lustre: Echo OBD driver; http://www.lustre.org/ [ 150.837651] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 184.305319] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 194.485125] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 194.515299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 195.758947] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 195.787280] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 195.836940] Lustre: lustre-MDT0000: new disk, initializing [ 195.938708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 195.952716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 199.005312] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 208.786618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 208.851779] Lustre: 6509:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 208.878871] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 208.882902] Lustre: Skipped 1 previous similar message [ 208.952683] Lustre: lustre-MDT0001: new disk, initializing [ 208.987862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 209.009746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 209.021207] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 212.130988] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.222991] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 223.247052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 223.439948] Lustre: lustre-OST0000: new disk, initializing [ 223.443262] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 223.496985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 227.449404] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.875136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 228.884137] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 228.928558] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 237.552088] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 237.656871] Lustre: lustre-OST0001: new disk, initializing [ 237.662389] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 237.722838] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 242.012784] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 244.745080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 244.753404] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 244.805361] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 251.671926] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 257.336582] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 267.084088] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing check_logdir /tmp/testlogs/ [ 270.500743] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing yml_node [ 273.952384] Lustre: DEBUG MARKER: Client: 2.17.52.4 [ 275.820024] Lustre: DEBUG MARKER: MDS: 2.17.52.4 [ 277.652535] Lustre: DEBUG MARKER: OSS: 2.17.52.4 [ 278.818173] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat May 9 13:19:16 EDT 2026 [ 289.766787] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 290.660362] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 306.148923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 306.148942] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 306.149974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 306.167537] Lustre: Skipped 3 previous similar messages [ 310.226305] Lustre: server umount lustre-MDT0000 complete [ 312.910519] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778347191 with bad export cookie 18078410895530528933 [ 312.914530] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 312.919831] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 316.384181] LustreError: 9886:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 316.384698] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 316.386728] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 316.386734] Lustre: Skipped 3 previous similar messages [ 316.397161] LustreError: 9886:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 316.414774] Lustre: Skipped 1 previous similar message [ 317.409852] LustreError: 6516:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 317.410277] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 317.424761] LustreError: 6516:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 317.428645] Lustre: Skipped 2 previous similar messages [ 319.224770] Lustre: server umount lustre-MDT0001 complete [ 322.293960] Lustre: server umount lustre-OST0000 complete [ 325.352139] Lustre: server umount lustre-OST0001 complete [ 330.835732] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 338.655436] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 365.377684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 365.532414] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 365.563396] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 365.617592] Lustre: lustre-MDT0000: new disk, initializing [ 365.672720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 365.687630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 368.215904] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 376.183335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 376.256491] Lustre: 14979:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 376.299515] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 376.303981] Lustre: Skipped 1 previous similar message [ 376.385196] Lustre: lustre-MDT0001: new disk, initializing [ 376.450659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 376.481228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 376.488606] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 379.147635] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 384.505432] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 387.443344] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 392.524416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 392.772489] Lustre: lustre-OST0000: new disk, initializing [ 392.781772] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 392.869055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 394.443674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 394.451901] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 394.524226] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 397.050631] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 402.292974] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 404.582960] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 404.685847] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 406.494804] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 406.599356] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 410.083624] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 410.085182] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 410.088682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 410.108849] Lustre: Skipped 1 previous similar message [ 413.655026] Lustre: server umount lustre-OST0000 complete [ 415.200301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 415.200858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 415.205012] Lustre: Skipped 1 previous similar message [ 415.207359] Lustre: Skipped 1 previous similar message [ 421.354702] Lustre: server umount lustre-MDT0000 complete [ 423.243597] LustreError: 14974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778347302 with bad export cookie 18078410895530530473 [ 423.252204] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 423.256316] LustreError: 14974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 423.505472] Lustre: server umount lustre-MDT0001 complete [ 425.897543] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 13:21:43 (1778347303) [ 427.330130] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 428.518147] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 13:21:46 (1778347306) [ 432.497041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 432.748682] LustreError: 18980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 432.806685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 434.962202] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 438.241429] LustreError: 18981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 439.574891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 441.891882] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 444.284988] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 445.923023] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 448.757255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 451.382131] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 453.710698] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 462.306583] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 462.307386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 462.314874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 462.317563] LustreError: Skipped 1 previous similar message [ 462.328822] Lustre: Skipped 3 previous similar messages [ 468.266728] Lustre: server umount lustre-OST0000 complete [ 470.496137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 470.500846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 470.507183] Lustre: Skipped 1 previous similar message [ 476.468920] Lustre: server umount lustre-MDT0000 complete [ 477.666297] LustreError: 18980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 477.678173] LustreError: 18980:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 478.361485] LustreError: 18961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778347357 with bad export cookie 18078410895530531355 [ 478.363228] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 478.377218] LustreError: 18961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 478.554958] Lustre: server umount lustre-MDT0001 complete [ 483.711503] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 485.208575] Key type lgssc unregistered [ 485.401062] LNet: 22155:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 485.406230] LNetError: 22155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 485.420725] LNet: Removed LNI 192.168.204.140@tcp [ 485.889214] Key type .llcrypt unregistered [ 485.890357] Key type ._llcrypt unregistered [ 496.658891] Key type ._llcrypt registered [ 496.662357] Key type .llcrypt registered [ 496.709303] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 497.102192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 497.140349] alg: No test for adler32 (adler32-zlib) [ 498.007239] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 498.103489] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 499.687159] Key type lgssc registered [ 500.091315] Lustre: Echo OBD driver; http://www.lustre.org/ [ 503.103106] loop: module loaded [ 503.128197] loop0: detected capacity change from 0 to 204800000 [ 503.213417] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 503.218456] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 503.344530] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 505.347938] loop0: detected capacity change from 0 to 204800000 [ 505.410269] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 505.414906] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 505.420392] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 507.119511] loop0: detected capacity change from 0 to 204800000 [ 509.031385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 509.039222] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 510.124685] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 510.140264] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 510.179234] Lustre: lustre-MDT0000: new disk, initializing [ 510.207969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 510.214405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 511.506048] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 515.154114] loop1: detected capacity change from 0 to 204800000 [ 517.096527] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 517.208321] Lustre: lustre-OST07c6: new disk, initializing [ 517.210772] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 517.213412] Lustre: Skipped 1 previous similar message [ 517.239727] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 519.307513] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 521.724113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 521.727593] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 521.738291] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 526.917085] Lustre: 25300:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 526.920850] Lustre: 25300:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 526.923186] Lustre: 25300:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 526.926278] Lustre: 25300:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 526.928973] Lustre: 25300:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 526.931770] Lustre: 25300:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 538.234054] Lustre: 25300:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 538.237611] Lustre: 25300:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 538.239939] Lustre: 25300:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 538.242954] Lustre: 25300:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 538.246753] Lustre: 25300:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 538.249846] Lustre: 25300:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 541.873639] Lustre: 25300:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 541.877772] Lustre: 25300:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 1 previous similar message [ 541.880669] Lustre: 25300:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 541.883494] Lustre: 25300:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 541.885977] Lustre: 25300:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 541.888208] Lustre: 25300:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 541.890851] Lustre: 25300:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 541.894787] Lustre: 25300:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 541.898124] Lustre: 25300:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 541.901655] Lustre: 25300:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 541.905612] Lustre: 25300:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 541.909915] Lustre: 25300:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 544.984027] Lustre: 24295:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 544.988351] Lustre: 24295:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 1 previous similar message [ 544.991780] Lustre: 24295:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 544.994085] Lustre: 24295:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 544.997364] Lustre: 24295:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 545.001599] Lustre: 24295:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 545.005751] Lustre: 24295:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 545.009295] Lustre: 24295:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 545.012438] Lustre: 24295:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 545.016696] Lustre: 24295:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 545.020322] Lustre: 24295:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 545.024136] Lustre: 24295:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 551.632620] Lustre: 24296:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 551.636710] Lustre: 24296:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 3 previous similar messages [ 551.639621] Lustre: 24296:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 551.643243] Lustre: 24296:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 551.647066] Lustre: 24296:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 551.650462] Lustre: 24296:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 551.654256] Lustre: 24296:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 551.658126] Lustre: 24296:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 551.661337] Lustre: 24296:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 551.664710] Lustre: 24296:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 551.667958] Lustre: 24296:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 551.671092] Lustre: 24296:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 561.619956] Lustre: 24325:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 561.623135] Lustre: 24325:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 5 previous similar messages [ 561.625746] Lustre: 24325:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 561.628681] Lustre: 24325:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 561.631137] Lustre: 24325:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 561.634227] Lustre: 24325:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 561.636457] Lustre: 24325:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 561.639247] Lustre: 24325:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 561.641319] Lustre: 24325:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 561.643252] Lustre: 24325:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 561.645744] Lustre: 24325:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 561.648186] Lustre: 24325:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 603.616399] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 603.616528] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 603.623478] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 608.647158] Lustre: server umount lustre-OST07c6 complete [ 611.494407] Lustre: server umount lustre-MDT0000 complete [ 616.644757] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 617.551819] Key type lgssc unregistered [ 617.681243] LNet: 27829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 617.683223] LNetError: 27829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 617.695275] LNet: Removed LNI 192.168.204.140@tcp [ 617.946814] Key type .llcrypt unregistered [ 617.948041] Key type ._llcrypt unregistered [ 625.085484] Key type ._llcrypt registered [ 625.087025] Key type .llcrypt registered [ 625.125401] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 625.574158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 625.585661] alg: No test for adler32 (adler32-zlib) [ 626.433958] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 626.513328] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 628.087100] Key type lgssc registered [ 628.404752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 630.684322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 630.688852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 631.834062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 632.932431] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 635.526877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 635.644944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 636.832560] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 638.457312] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 639.548240] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 641.625886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 641.722418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 642.792728] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 643.591245] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 645.352090] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 651.236680] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 651.237093] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 651.241771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 651.247414] Lustre: Skipped 1 previous similar message [ 656.354076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 656.355847] Lustre: Skipped 1 previous similar message [ 656.714063] Lustre: server umount lustre-OST0000 complete [ 661.472756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 661.472818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 661.473427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 661.478433] Lustre: Skipped 2 previous similar messages [ 666.592111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 666.594020] Lustre: Skipped 2 previous similar messages [ 671.712207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 671.714355] Lustre: Skipped 1 previous similar message [ 672.223093] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 672.287273] Lustre: server umount lustre-MDT0000 complete [ 673.527234] LustreError: 29042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778347552 with bad export cookie 9555801108033253594 [ 673.530185] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 673.532727] LustreError: 29042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 673.646383] Lustre: server umount lustre-MDT0001 complete [ 677.196247] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 678.235414] Key type lgssc unregistered [ 678.361328] LNet: 32443:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 678.363987] LNetError: 32443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 678.372365] LNet: Removed LNI 192.168.204.140@tcp [ 678.659200] Key type .llcrypt unregistered [ 678.660298] Key type ._llcrypt unregistered [ 685.802793] Key type ._llcrypt registered [ 685.803812] Key type .llcrypt registered [ 685.840747] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 686.282680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 686.292346] alg: No test for adler32 (adler32-zlib) [ 687.148872] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 687.233440] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 688.815104] Key type lgssc registered [ 689.175106] Lustre: Echo OBD driver; http://www.lustre.org/ [ 691.725836] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 691.734271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 692.888633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 694.176148] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 696.989076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 697.100977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 698.393581] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 700.165288] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 701.368572] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 703.671063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 703.791330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 705.631672] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 707.450837] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 717.095034] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 722.403064] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 722.403295] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 722.406888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 722.410382] Lustre: Skipped 1 previous similar message [ 723.728418] Lustre: server umount lustre-OST0000 complete [ 727.126552] Lustre: server umount lustre-MDT0000 complete [ 727.520598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 727.521452] LustreError: 35826:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 727.526510] Lustre: Skipped 1 previous similar message [ 727.534508] LustreError: 35826:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 728.416174] LustreError: 33653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778347607 with bad export cookie 367915448966696860 [ 728.419198] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 728.419848] LustreError: 33653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 728.538814] Lustre: server umount lustre-MDT0001 complete [ 732.359622] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 733.422826] Key type lgssc unregistered [ 733.545511] LNet: 36869:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 733.548885] LNetError: 36869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 733.559266] LNet: Removed LNI 192.168.204.140@tcp [ 733.853774] Key type .llcrypt unregistered [ 733.855759] Key type ._llcrypt unregistered [ 742.118605] Key type ._llcrypt registered [ 742.120042] Key type .llcrypt registered [ 742.155137] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 742.435872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 742.473303] alg: No test for adler32 (adler32-zlib) [ 743.311274] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 743.381764] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 744.951123] Key type lgssc registered [ 745.248930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 747.494364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 747.500960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 748.637812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 749.754554] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 752.227664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 752.329579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 753.467520] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 755.065530] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 756.265117] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 758.296731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 758.397180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 759.460951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 760.024423] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 761.541912] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 767.969494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 767.969984] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 767.970938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 767.973784] Lustre: Skipped 1 previous similar message [ 770.834019] Lustre: server umount lustre-MDT0000 complete [ 771.953042] LustreError: 38224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778347651 with bad export cookie 1901587574738894356 [ 771.955431] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 771.957716] LustreError: 38224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 773.087781] LustreError: 39857:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 773.087949] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 773.092682] LustreError: 39857:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 773.101240] Lustre: Skipped 1 previous similar message [ 773.103473] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 773.104935] Lustre: Skipped 3 previous similar messages [ 776.095821] LustreError: 38240:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 776.096057] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 776.103316] LustreError: 38240:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 776.104691] Lustre: Skipped 1 previous similar message [ 778.243925] LustreError: 38241:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 778.244527] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.40@tcp (stopping) [ 781.279569] LustreError: 38239:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 782.304141] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 782.305642] Lustre: Skipped 1 previous similar message [ 786.399084] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 786.466727] Lustre: server umount lustre-MDT0001 complete [ 798.208599] Lustre: server umount lustre-OST0000 complete [ 803.365545] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 804.379856] Key type lgssc unregistered [ 804.504417] LNet: 41748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 804.508691] LNetError: 41748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 804.515315] LNet: Removed LNI 192.168.204.140@tcp [ 804.807966] Key type .llcrypt unregistered [ 804.808982] Key type ._llcrypt unregistered [ 813.499117] Key type ._llcrypt registered [ 813.500214] Key type .llcrypt registered [ 813.540427] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 813.882810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 813.940037] alg: No test for adler32 (adler32-zlib) [ 814.792837] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 814.877584] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 816.463153] Key type lgssc registered [ 816.834332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 819.440702] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 819.448273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 820.605572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 821.880632] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 824.626156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 824.749354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 826.033566] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 827.865507] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 829.083135] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 831.297084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 831.408292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 833.214345] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 834.997093] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 845.733588] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 855.007739] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 855.010077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 855.013928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 855.562739] Lustre: lustre-OST0000: Not available for connect from 192.168.204.40@tcp (stopping) [ 856.033555] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 856.039720] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 858.703408] Lustre: server umount lustre-OST0000 complete [ 871.233543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 871.234362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 871.238575] Lustre: Skipped 1 previous similar message [ 871.241076] Lustre: Skipped 2 previous similar messages [ 876.512044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 876.513988] Lustre: Skipped 2 previous similar messages [ 877.399187] Lustre: server umount lustre-MDT0000 complete [ 878.514971] LustreError: 43106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778347757 with bad export cookie 10355629136509562268 [ 878.517870] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 878.518331] LustreError: 43106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 884.712322] Lustre: server umount lustre-MDT0001 complete [ 888.088269] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 889.100803] Key type lgssc unregistered [ 889.226329] LNet: 46502:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 889.229234] LNetError: 46502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 889.241294] LNet: Removed LNI 192.168.204.140@tcp [ 889.551117] Key type .llcrypt unregistered [ 889.552241] Key type ._llcrypt unregistered [ 898.175115] Key type ._llcrypt registered [ 898.176279] Key type .llcrypt registered [ 898.211873] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 898.534026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 898.561253] alg: No test for adler32 (adler32-zlib) [ 899.406495] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 899.488240] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 901.071100] Key type lgssc registered [ 901.398722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 903.850250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 903.856930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 905.006915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 906.296537] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 909.033939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 909.149384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 910.402760] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 912.096455] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 913.229712] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 915.414794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 915.526575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 917.322114] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 918.987979] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 930.914445] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 936.459489] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 937.490727] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 948.187740] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-05-0913h30m26s [ 948.661159] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 949.769907] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.40@tcp (stopping) [ 950.240021] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 950.243602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 950.787854] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.40@tcp (stopping) [ 950.790642] Lustre: Skipped 3 previous similar messages [ 955.543339] Lustre: server umount lustre-MDT0000 complete [ 956.591802] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 956.931532] LustreError: 47876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 956.939169] LustreError: 47876:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 958.706199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 958.743856] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 958.828576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 958.857248] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 959.846210] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 959.995595] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 961.074114] Lustre: DEBUG MARKER: Wait for df (35961) ... [ 961.610252] Lustre: DEBUG MARKER: done [ 964.064796] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 964.068237] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 964.073530] Lustre: Skipped 2 previous similar messages [ 964.075965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 964.079488] Lustre: Skipped 3 previous similar messages [ 965.092569] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 968.647401] Lustre: server umount lustre-OST0000 complete [ 970.209022] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 970.209042] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 970.209727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 970.209731] Lustre: Skipped 2 previous similar messages [ 970.213265] Lustre: Skipped 1 previous similar message [ 975.957935] Lustre: server umount lustre-MDT0000 complete [ 976.288264] LustreError: 48591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 976.292798] LustreError: 48591:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 977.210176] LustreError: 50194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778347856 with bad export cookie 16864917943756015391 [ 977.213862] LustreError: 50194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 977.213917] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 977.336661] Lustre: server umount lustre-MDT0001 complete [ 980.957123] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 981.931955] Key type lgssc unregistered [ 982.064362] LNet: 52451:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.068108] LNetError: 52451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 982.077326] LNet: Removed LNI 192.168.204.140@tcp [ 982.350880] Key type .llcrypt unregistered [ 982.351962] Key type ._llcrypt unregistered [ 995.313682] Key type ._llcrypt registered [ 995.314772] Key type .llcrypt registered [ 995.355897] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 995.726745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 995.754170] alg: No test for adler32 (adler32-zlib) [ 996.611549] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 996.700969] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 998.287121] Key type lgssc registered [ 998.635974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1001.140726] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1001.147573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1002.244500] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1002.251958] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1002.314400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1003.647182] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1006.284187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1006.315263] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1006.324964] Lustre: 54619:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1006.418246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1007.584391] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1009.174533] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1010.185906] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1012.172742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1012.248971] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1012.314462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1014.114470] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1015.816520] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1017.860594] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1024.512469] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-05-0913h31m42s [ 1024.968175] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1027.209768] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1028.000860] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1028.017556] LustreError: 54623:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1048.084365] Lustre: lustre-MDT0000: Client 10b45256-e9a8-4717-836d-e57190234a61 (at 192.168.204.40@tcp) reconnecting [ 1068.063071] LustreError: 54623:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1068.739636] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1069.495046] Lustre: DEBUG MARKER: done [ 1074.145689] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1074.146322] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1074.155381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1074.157773] LustreError: Skipped 1 previous similar message [ 1077.067932] Lustre: server umount lustre-OST0000 complete [ 1079.265655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1079.266782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1079.269273] Lustre: Skipped 2 previous similar messages [ 1079.271012] Lustre: Skipped 1 previous similar message [ 1080.288948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1080.290835] Lustre: Skipped 2 previous similar messages [ 1080.418230] Lustre: server umount lustre-MDT0000 complete [ 1081.620306] LustreError: 54610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778347960 with bad export cookie 12481297222157912939 [ 1081.623246] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1081.624151] LustreError: 54610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1087.790767] Lustre: server umount lustre-MDT0001 complete [ 1091.132234] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 1092.090525] Key type lgssc unregistered [ 1092.215395] LNet: 58580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1092.219566] LNetError: 58580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1092.227326] LNet: Removed LNI 192.168.204.140@tcp [ 1092.517104] Key type .llcrypt unregistered [ 1092.518173] Key type ._llcrypt unregistered [ 1107.674529] Key type ._llcrypt registered [ 1107.675593] Key type .llcrypt registered [ 1107.712296] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 1108.117557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1108.162067] alg: No test for adler32 (adler32-zlib) [ 1109.015813] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 1109.101181] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 1110.679215] Key type lgssc registered [ 1111.010905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1112.014644] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1112.022067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1113.109993] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1113.117224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1113.173151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1113.487596] Lustre: Failing over lustre-MDT0000 [ 1113.585490] Lustre: server umount lustre-MDT0000 complete [ 1115.504418] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 13:33:13 (1778347993) [ 1117.549056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1117.696861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1118.950100] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1121.630222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1121.663376] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1121.674318] Lustre: 61339:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1121.767444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1122.991270] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1124.750963] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1125.903583] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1128.055943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1128.122049] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1128.185077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1129.961673] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1131.723972] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1134.604564] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1140.010168] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1142.239909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1142.242763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1142.247837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1144.800262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1144.801206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1144.805611] Lustre: Skipped 1 previous similar message [ 1147.734990] Lustre: server umount lustre-MDT0000 complete [ 1148.864090] LustreError: 61330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778348027 with bad export cookie 8606697834184860475 [ 1148.867373] LustreError: 61330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1148.867524] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1148.972450] Lustre: server umount lustre-MDT0001 complete [ 1149.991089] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1151.378472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1154.037510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1158.047537] LustreError: 60262:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000941080ed x1864733019897088/t0(0) o250->MGC192.168.204.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1158.183743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1158.215241] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1159.416462] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1162.123081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1162.206698] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1163.421227] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1164.322528] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1165.021540] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1165.343082] Lustre: 60264:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778348028/real 1778348028] req@000000007ef19da0 x1864733019896704/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778348044 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1166.184561] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1169.377281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1169.378357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1169.380875] Lustre: Skipped 2 previous similar messages [ 1169.382492] Lustre: Skipped 3 previous similar messages [ 1170.399147] Lustre: 60265:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778348033/real 1778348033] req@0000000056da4dfa x1864733019896832/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778348049 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1171.423125] Lustre: 60266:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778348034/real 1778348034] req@00000000254bff90 x1864733019896960/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778348050 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1173.971917] Lustre: server umount lustre-MDT0000 complete [ 1174.495093] Lustre: 60263:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778348037/real 1778348037] req@000000007d4f733b x1864733019897216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778348053 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1174.498800] LustreError: 66029:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1174.510419] LustreError: 66029:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1175.098043] LustreError: 61331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778348054 with bad export cookie 8606697834184862743 [ 1175.100932] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1175.101346] LustreError: 61331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1175.193580] Lustre: server umount lustre-MDT0001 complete [ 1177.159859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1180.056919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1185.759650] LustreError: 60262:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000eb7ff0a8 x1864733019916288/t0(0) o250->MGC192.168.204.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1185.872569] LustreError: 62957:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1185.901261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1185.903352] Lustre: Skipped 1 previous similar message [ 1185.926715] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1187.097847] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1189.871356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1189.956364] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1191.441138] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1193.163171] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1194.293099] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1195.103127] Lustre: 60265:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778348058/real 1778348058] req@00000000941080ed x1864733019915904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778348074 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1195.491859] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1195.493631] Lustre: Skipped 1 previous similar message [ 1199.588822] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1204.705369] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1204.706438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1204.709222] Lustre: Skipped 1 previous similar message [ 1207.192988] Lustre: server umount lustre-MDT0000 complete [ 1208.392265] LustreError: 61369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778348087 with bad export cookie 8606697834184864122 [ 1208.395387] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1208.399394] LustreError: 61369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1208.513041] Lustre: server umount lustre-MDT0001 complete [ 1210.619909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1211.706954] Lustre: DEBUG MARKER: files compared the same [ 1222.629798] Lustre: server umount lustre-OST0000 complete [ 1228.396745] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 1229.363025] Key type lgssc unregistered [ 1229.489376] LNet: 70250:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1229.493236] LNetError: 70250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1229.501356] LNet: Removed LNI 192.168.204.140@tcp [ 1229.782624] Key type .llcrypt unregistered [ 1229.784724] Key type ._llcrypt unregistered [ 1239.095520] Key type ._llcrypt registered [ 1239.096587] Key type .llcrypt registered [ 1239.132604] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 1239.425912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1239.437192] alg: No test for adler32 (adler32-zlib) [ 1240.283556] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 1240.361674] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 1241.935140] Key type lgssc registered [ 1242.293042] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1244.970180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1244.976781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1246.129169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1247.430676] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug +malloc all [ 1250.307734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1250.419571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1251.800603] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug +malloc all [ 1253.861465] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1255.059626] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1257.237036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1257.328385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1259.103294] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug +malloc all [ 1260.933714] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1271.649667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1280.991885] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1280.997734] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1281.011372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1282.016542] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1282.017583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1282.017833] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1282.036915] Lustre: Skipped 1 previous similar message [ 1285.128146] Lustre: server umount lustre-OST0000 complete [ 1286.623696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1286.627766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1286.633938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1292.255763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1292.259947] Lustre: Skipped 3 previous similar messages [ 1292.518580] Lustre: server umount lustre-MDT0000 complete [ 1293.879636] LustreError: 71608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778348172 with bad export cookie 10794416726681962785 [ 1293.881039] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1293.882982] LustreError: 71608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1293.988684] Lustre: server umount lustre-MDT0001 complete [ 1297.735249] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 1298.755958] Key type lgssc unregistered [ 1298.896340] LNet: 74826:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1298.898715] LNetError: 74826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1298.907368] LNet: Removed LNI 192.168.204.140@tcp [ 1299.199049] Key type .llcrypt unregistered [ 1299.200416] Key type ._llcrypt unregistered [ 1308.718213] Key type ._llcrypt registered [ 1308.719841] Key type .llcrypt registered [ 1308.754600] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 1309.149931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1309.168352] alg: No test for adler32 (adler32-zlib) [ 1310.019111] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 1310.098258] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 1311.671107] Key type lgssc registered [ 1312.025080] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1314.635517] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1314.644795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1330.079384] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1335.199140] LustreError: 76177:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.140@tcp: failed processing log, type 4: rc = -110 [ 1361.823239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1363.726503] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1365.500494] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1368.073974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1368.121707] Lustre: *** cfs_fail_loc=706, val=0*** [ 1368.260226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1368.281877] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1369.568830] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1372.521153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1372.645037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1373.897101] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1375.595880] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1376.796998] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1377.760743] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1377.763093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1377.767155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1378.784257] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1378.786878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1378.790704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1379.536868] Lustre: server umount lustre-OST0000 complete [ 1381.857550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1381.858891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1381.861915] Lustre: Skipped 1 previous similar message [ 1386.976030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1386.978310] Lustre: Skipped 2 previous similar messages [ 1392.095987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1392.098016] Lustre: Skipped 1 previous similar message [ 1395.167074] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1395.223094] Lustre: server umount lustre-MDT0000 complete [ 1396.193204] LustreError: 77152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1396.199166] LustreError: 77152:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1396.414720] LustreError: 76184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778348275 with bad export cookie 4949931568053619039 [ 1396.420640] LustreError: 76184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1396.420753] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1396.549466] Lustre: server umount lustre-MDT0001 complete [ 1400.060510] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 1401.158722] Key type lgssc unregistered [ 1401.286249] LNet: 79437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1401.289616] LNetError: 79437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1401.301455] LNet: Removed LNI 192.168.204.140@tcp [ 1401.608066] Key type .llcrypt unregistered [ 1401.609249] Key type ._llcrypt unregistered [ 1410.661281] Key type ._llcrypt registered [ 1410.662397] Key type .llcrypt registered [ 1410.697420] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 1411.006988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1411.038156] alg: No test for adler32 (adler32-zlib) [ 1411.880591] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 1411.959129] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 1413.535170] Key type lgssc registered [ 1413.900781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1416.841409] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1416.852602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1419.187112] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1422.021767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1422.199227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1423.527821] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1426.150897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1426.252972] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1428.278645] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1431.174039] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1431.179392] Lustre: MGS: Client c719ae6d-4499-47fd-9bdd-3c7fc4faa718 (at 0@lo) reconnecting [ 1431.186966] LustreError: 81495:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1431.262435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1431.290353] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1432.514602] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1441.760911] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1441.766280] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1441.776140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1442.786697] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1442.788082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1442.790169] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1442.800654] Lustre: Skipped 1 previous similar message [ 1446.609358] Lustre: server umount lustre-OST0000 complete [ 1453.024353] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1453.024928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1453.032271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1453.034134] Lustre: Skipped 1 previous similar message [ 1454.271711] Lustre: server umount lustre-MDT0000 complete [ 1455.605984] Lustre: server umount lustre-MDT0001 complete [ 1456.823093] Lustre: server umount MGS complete [ 1460.834234] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 13:38:59 (1778348339) [ 1470.182596] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 1473.017813] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 1485.264053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1486.464286] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1490.533991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1490.614492] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1490.629735] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1490.670021] Lustre: lustre-MDT0001: new disk, initializing [ 1490.695686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1490.706155] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1490.711672] Lustre: 88543:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1490.718809] LustreError: 88543:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1492.229284] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1497.072107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1497.165499] Lustre: lustre-OST0000: new disk, initializing [ 1497.167458] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1497.186960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1499.004774] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1502.231245] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1502.687819] Lustre: 89840:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000004b394d66 x1864733337617920/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1778348397 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1504.810764] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1504.820616] Lustre: MGS: Client 870bf0ad-b825-4c6c-b86e-1170f19df515 (at 0@lo) reconnecting [ 1504.846476] Lustre: 90223:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1504.861143] LustreError: 88529:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1504.866723] LustreError: 88529:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1504.875139] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1504.926343] Lustre: lustre-MDT0000: new disk, initializing [ 1504.997307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1506.016114] Lustre: 90250:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000bc0fdd48 x1864733337623296/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1778348401 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1506.080820] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1506.638578] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1511.778131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1511.780826] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1515.106042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 1515.109053] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 1515.170868] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1535.968086] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1535.970971] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1535.975139] Lustre: Skipped 1 previous similar message [ 1535.977369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1541.773492] Lustre: server umount lustre-OST0000 complete [ 1547.233118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1547.233448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1547.234160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1547.234164] Lustre: Skipped 2 previous similar messages [ 1547.245602] Lustre: Skipped 2 previous similar messages [ 1549.258921] Lustre: server umount lustre-MDT0000 complete [ 1556.742627] Lustre: server umount lustre-MDT0001 complete [ 1558.469743] Lustre: server umount MGS complete [ 1562.586177] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 13:40:40 (1778348440) [ 1569.331590] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 1570.523589] Key type lgssc unregistered [ 1570.654480] LNet: 92716:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1570.657335] LNetError: 92716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1570.666514] LNet: Removed LNI 192.168.204.140@tcp [ 1571.030799] Key type .llcrypt unregistered [ 1571.032034] Key type ._llcrypt unregistered [ 1578.934144] Key type ._llcrypt registered [ 1578.935256] Key type .llcrypt registered [ 1578.970970] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 1579.273882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1579.312043] alg: No test for adler32 (adler32-zlib) [ 1580.159381] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 1580.240826] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 1581.815158] Key type lgssc registered [ 1582.154446] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1584.649224] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1584.654937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1585.814060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1587.137510] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1590.081317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1590.206770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1591.644772] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1593.548403] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1594.800726] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1597.017032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1597.150981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1599.064249] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1600.867765] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1610.470846] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 1615.846526] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1615.848241] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1615.852248] LustreError: Skipped 1 previous similar message [ 1615.861512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1615.866986] Lustre: Skipped 1 previous similar message [ 1616.151605] Lustre: server umount lustre-OST0000 complete [ 1619.943094] Lustre: server umount lustre-MDT0000 complete [ 1620.960870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1620.961128] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1620.961491] LustreError: 93899:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1620.961498] LustreError: 93899:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1620.964561] Lustre: Skipped 2 previous similar messages [ 1621.232154] LustreError: 93879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778348500 with bad export cookie 9564501221336418684 [ 1621.235057] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1621.236060] LustreError: 93879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1621.340827] Lustre: server umount lustre-MDT0001 complete [ 1625.295540] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 1626.299252] Key type lgssc unregistered [ 1626.435422] LNet: 97099:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1626.439655] LNetError: 97099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1626.450597] LNet: Removed LNI 192.168.204.140@tcp [ 1626.795574] Key type .llcrypt unregistered [ 1626.796817] Key type ._llcrypt unregistered [ 1633.817085] Key type ._llcrypt registered [ 1633.818411] Key type .llcrypt registered [ 1633.855716] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 1634.171238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1634.228203] alg: No test for adler32 (adler32-zlib) [ 1635.078263] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 1635.157315] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 1636.735107] Key type lgssc registered [ 1637.089653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1639.233226] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1639.235491] LustreError: 98142:0:(tgt_mount.c:2501:server_fill_super()) cfs_race id 716 sleeping [ 1639.237783] LustreError: 98143:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 waking [ 1639.240930] LustreError: 98142:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 1639.245300] LustreError: 98142:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1639.248482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1639.249346] LustreError: 98142:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1640.403655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1643.434818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1643.546487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1644.860536] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1646.840819] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing lsmod [ 1647.874004] LustreError: 99314:0:(tgt_mount.c:2501:server_fill_super()) cfs_race id 716 sleeping [ 1647.874049] LustreError: 99313:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 waking [ 1647.880793] LustreError: 99314:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 1647.884304] LustreError: 99314:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1647.887334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1647.889678] LustreError: 99314:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1648.013437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1649.191531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1649.194425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1649.200247] Lustre: Skipped 1 previous similar message [ 1649.206829] Lustre: Skipped 2 previous similar messages [ 1653.282551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1653.287151] Lustre: Skipped 2 previous similar messages [ 1655.343607] Lustre: server umount lustre-MDT0000 complete [ 1656.779138] LustreError: 98617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778348535 with bad export cookie 7916552829597090620 [ 1656.782384] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1656.785051] LustreError: 98617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1657.335311] LustreError: 98610:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000003d5a8383 x1864733571431936/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1657.432148] Lustre: server umount lustre-MDT0001 complete [ 1669.300996] Lustre: server umount lustre-OST0000 complete [ 1672.591280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1672.754120] LustreError: 100378:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1672.762350] LustreError: 100378:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1672.777335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1674.062668] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1676.986249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1677.098089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1678.306056] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1679.976724] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1681.134703] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1683.277306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1685.274208] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1686.924655] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1697.766851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 1708.002885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1708.003221] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1708.004096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1708.006515] Lustre: Skipped 2 previous similar messages [ 1708.008945] LustreError: Skipped 1 previous similar message [ 1709.735813] Lustre: server umount lustre-OST0000 complete [ 1713.120647] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1713.120848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1713.121572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1713.121577] Lustre: Skipped 1 previous similar message [ 1713.131413] Lustre: Skipped 1 previous similar message [ 1717.205845] Lustre: server umount lustre-MDT0000 complete [ 1718.239952] LustreError: 101099:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1718.244325] LustreError: 101099:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1718.588990] LustreError: 100359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778348597 with bad export cookie 7916552829597091278 [ 1718.591946] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1718.592074] LustreError: 100359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1718.703708] Lustre: server umount lustre-MDT0001 complete [ 1722.690149] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 1723.651744] Key type lgssc unregistered [ 1723.774382] LNet: 103575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1723.776706] LNetError: 103575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1723.783335] LNet: Removed LNI 192.168.204.140@tcp [ 1724.083667] Key type .llcrypt unregistered [ 1724.084756] Key type ._llcrypt unregistered [ 1733.087918] Key type ._llcrypt registered [ 1733.089069] Key type .llcrypt registered [ 1733.123480] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 1733.465311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1733.509708] alg: No test for adler32 (adler32-zlib) [ 1734.357782] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 1734.436656] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 1736.015125] Key type lgssc registered [ 1736.341229] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1738.912723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1738.925368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1740.100572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1741.496244] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1744.337256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1744.452414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1745.755788] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1747.580136] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1748.875358] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1751.288984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1751.418568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1753.386033] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1755.407484] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1765.800494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 1767.761776] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1768.531144] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1769.952840] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1769.957720] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1769.965459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1770.978935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1770.979968] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1770.980331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1775.567570] Lustre: server umount lustre-OST0000 complete [ 1780.192184] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1780.195468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1780.200719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1780.204699] Lustre: Skipped 1 previous similar message [ 1782.929102] Lustre: server umount lustre-MDT0000 complete [ 1784.050957] LustreError: 104930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778348663 with bad export cookie 1057935800782530345 [ 1784.053788] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1784.054460] LustreError: 104930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1784.160815] Lustre: server umount lustre-MDT0001 complete [ 1788.120562] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 1789.279997] Key type lgssc unregistered [ 1789.438686] LNet: 108251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1789.445946] LNetError: 108251:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1789.462679] LNet: Removed LNI 192.168.204.140@tcp [ 1789.932924] Key type .llcrypt unregistered [ 1789.934429] Key type ._llcrypt unregistered [ 1798.038883] Key type ._llcrypt registered [ 1798.040796] Key type .llcrypt registered [ 1798.103267] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 1798.528744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1798.535959] alg: No test for adler32 (adler32-zlib) [ 1799.377556] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 1799.456123] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 1801.031194] Key type lgssc registered [ 1801.540815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1804.197168] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1804.207047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1805.386138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1806.768670] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1809.450190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1809.579113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1811.237556] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1813.569595] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1814.980935] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1817.380711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1817.499698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1819.542454] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1821.488490] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1831.849360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 1834.976599] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1834.978942] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1834.982604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1837.026799] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1837.027643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1837.030465] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1841.104953] Lustre: server umount lustre-OST0000 complete [ 1845.727755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1845.729940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1845.733645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1845.735750] Lustre: Skipped 1 previous similar message [ 1848.546336] Lustre: server umount lustre-MDT0000 complete [ 1850.150136] LustreError: 109415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778348729 with bad export cookie 4549842932841911224 [ 1850.153780] LustreError: 109415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1850.154059] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1850.269248] Lustre: server umount lustre-MDT0001 complete [ 1854.645419] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 1855.955290] Key type lgssc unregistered [ 1856.138600] LNet: 112633:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1856.143634] LNetError: 112633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1856.153581] LNet: Removed LNI 192.168.204.140@tcp [ 1856.595375] Key type .llcrypt unregistered [ 1856.597695] Key type ._llcrypt unregistered [ 1868.444133] Key type ._llcrypt registered [ 1868.445651] Key type .llcrypt registered [ 1868.488945] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 1868.849650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1868.897281] alg: No test for adler32 (adler32-zlib) [ 1869.777104] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 1869.874295] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 1871.447185] Key type lgssc registered [ 1871.988335] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1875.220106] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1875.231537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1876.421079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1877.629590] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1881.273773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1881.443594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1883.010218] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1884.841767] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1886.526982] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1889.243885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1889.377368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1891.160808] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1893.442993] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1901.672098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 1906.833605] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1908.092677] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1908.097017] Lustre: Skipped 2 previous similar messages [ 1909.992943] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1909.998175] Lustre: Skipped 2 previous similar messages [ 1912.848682] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1912.867424] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1920.744433] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.40@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1920.751651] Lustre: Skipped 2 previous similar messages [ 1928.207356] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.40@tcp 192.168.0.[2,10]@tcp [ 1930.335140] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.40/24@tcp in log lustre-MDT0000 [ 1930.340049] Lustre: Skipped 2 previous similar messages [ 1935.881474] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.40/24@tcp [ 1935.885232] Lustre: Skipped 1 previous similar message [ 1937.893674] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1937.894351] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1937.895847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1937.900035] Lustre: Skipped 1 previous similar message [ 1937.910592] LustreError: Skipped 1 previous similar message [ 1943.009234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1943.011111] Lustre: Skipped 2 previous similar messages [ 1943.592402] Lustre: server umount lustre-OST0000 complete [ 1948.130530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1948.131103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1948.131903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1948.146489] Lustre: Skipped 1 previous similar message [ 1953.249178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1953.254058] Lustre: Skipped 2 previous similar messages [ 1958.367965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1958.369941] Lustre: Skipped 1 previous similar message [ 1959.391082] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1959.447851] Lustre: server umount lustre-MDT0000 complete [ 1960.648450] LustreError: 115624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778348839 with bad export cookie 11920758588496779045 [ 1960.652097] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1960.655045] LustreError: 115624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1960.760595] Lustre: server umount lustre-MDT0001 complete [ 1965.111898] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 1966.206858] Key type lgssc unregistered [ 1966.381645] LNet: 118804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1966.386799] LNetError: 118804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1966.398543] LNet: Removed LNI 192.168.204.140@tcp [ 1966.840192] Key type .llcrypt unregistered [ 1966.844619] Key type ._llcrypt unregistered [ 1978.472028] Key type ._llcrypt registered [ 1978.473189] Key type .llcrypt registered [ 1978.509517] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 1979.048305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1979.059305] alg: No test for adler32 (adler32-zlib) [ 1979.909193] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 1979.997287] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 1981.575120] Key type lgssc registered [ 1981.914529] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1984.641502] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1984.649055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1985.805458] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1985.810098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1985.814403] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.40/24@tcp [ 1987.144610] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1990.771892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1990.893856] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1990.901883] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.40/24@tcp [ 1990.917401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1992.432321] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1994.465533] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1996.055932] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1999.086251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1999.268230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2001.834378] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2004.074787] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2010.541216] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 2020.836387] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2020.837619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2020.841449] LustreError: Skipped 1 previous similar message [ 2020.850353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2020.856465] Lustre: Skipped 1 previous similar message [ 2024.334617] Lustre: server umount lustre-OST0000 complete [ 2025.953551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2025.954746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2025.962334] Lustre: Skipped 2 previous similar messages [ 2031.072396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2031.075504] Lustre: Skipped 2 previous similar messages [ 2031.781642] Lustre: server umount lustre-MDT0000 complete [ 2033.148456] LustreError: 122849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778348912 with bad export cookie 17608706765594369047 [ 2033.150655] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2033.153453] LustreError: 122849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2033.282721] Lustre: server umount lustre-MDT0001 complete [ 2037.152963] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 2038.133055] Key type lgssc unregistered [ 2038.256426] LNet: 123582:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2038.259720] LNetError: 123582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2038.271275] LNet: Removed LNI 192.168.204.140@tcp [ 2038.579726] Key type .llcrypt unregistered [ 2038.581418] Key type ._llcrypt unregistered [ 2047.721271] Key type ._llcrypt registered [ 2047.722394] Key type .llcrypt registered [ 2047.771470] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 2054.433472] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 2054.856702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2054.866783] alg: No test for adler32 (adler32-zlib) [ 2055.715205] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 2055.795877] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 2057.367154] Key type lgssc registered [ 2057.719224] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2070.487436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2070.492163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2071.594945] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2071.612805] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2071.659065] Lustre: lustre-MDT0000: new disk, initializing [ 2071.685837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2071.693992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2073.501418] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2078.204231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2078.271338] Lustre: 126919:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2078.313328] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2078.317339] Lustre: Skipped 1 previous similar message [ 2078.366515] Lustre: lustre-MDT0001: new disk, initializing [ 2078.405131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2078.418732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2078.422702] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2079.839335] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2083.447225] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2084.616126] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2086.748927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2086.856516] Lustre: lustre-OST0000: new disk, initializing [ 2086.858699] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2086.890986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2088.693916] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2091.705379] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2092.538162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2092.541733] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2092.579329] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2093.077842] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2093.151919] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2094.443410] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2094.510579] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2097.632922] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2097.633202] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2097.636415] LustreError: Skipped 1 previous similar message [ 2097.640827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2097.643468] Lustre: Skipped 1 previous similar message [ 2101.201867] Lustre: server umount lustre-OST0000 complete [ 2102.752248] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2102.752924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2102.756093] Lustre: Skipped 2 previous similar messages [ 2102.760450] Lustre: Skipped 1 previous similar message [ 2107.872273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2107.874018] Lustre: Skipped 1 previous similar message [ 2108.562596] Lustre: server umount lustre-MDT0000 complete [ 2109.690887] LustreError: 127700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778348988 with bad export cookie 10244382145260315827 [ 2109.693678] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2109.693865] LustreError: 127700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2109.796027] Lustre: server umount lustre-MDT0001 complete [ 2112.433304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2112.597879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2113.812320] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2116.420664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2117.701226] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2119.299057] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2120.366508] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2122.403044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2122.502386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2122.504414] Lustre: Skipped 1 previous similar message [ 2124.156531] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2125.802566] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2127.055066] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2127.125077] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2128.343694] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2128.415269] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2131.618772] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2131.680339] Lustre: lustre-OST0001: new disk, initializing [ 2131.683189] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2131.710936] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2133.519838] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2134.269871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2134.273474] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2134.296694] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2136.730693] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2136.805773] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2138.157819] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2138.229374] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2139.458257] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff906908779800.ost_server_uuid 50 [ 2140.080032] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff906908779800.ost_server_uuid in FULL state after 0 sec [ 2144.737538] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2144.738719] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2144.740139] LustreError: Skipped 1 previous similar message [ 2144.745207] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2144.747693] Lustre: Skipped 1 previous similar message [ 2148.302719] Lustre: server umount lustre-OST0001 complete [ 2149.858064] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2149.858764] LustreError: 134165:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2149.864236] Lustre: Skipped 1 previous similar message [ 2149.864860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2149.871389] LustreError: 134165:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2154.463974] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2154.466922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2154.977270] LustreError: 134170:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2154.982247] LustreError: 134170:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2155.718894] Lustre: server umount lustre-OST0000 complete [ 2157.536088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2160.096869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2160.100166] Lustre: Skipped 4 previous similar messages [ 2171.359197] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2171.443762] Lustre: server umount lustre-MDT0000 complete [ 2172.689172] LustreError: 130707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778349051 with bad export cookie 10244382145260316681 [ 2172.692392] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2172.693129] LustreError: 130707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2172.816582] Lustre: server umount lustre-MDT0001 complete [ 2179.148861] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 2180.206924] Key type lgssc unregistered [ 2180.317287] LNet: 136557:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2180.320387] LNetError: 136557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2180.331314] LNet: Removed LNI 192.168.204.140@tcp [ 2180.587706] Key type .llcrypt unregistered [ 2180.588828] Key type ._llcrypt unregistered [ 2194.132509] Key type ._llcrypt registered [ 2194.133756] Key type .llcrypt registered [ 2194.173079] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 2200.323530] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 2200.638738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2200.692549] alg: No test for adler32 (adler32-zlib) [ 2201.542788] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 2201.624747] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 2203.207178] Key type lgssc registered [ 2203.763845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2216.609088] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2216.614742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2217.703437] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2217.713524] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2217.750369] Lustre: lustre-MDT0000: new disk, initializing [ 2217.772513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2217.778150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2219.115376] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2223.628911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2223.670400] Lustre: 140648:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2223.692846] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2223.695029] Lustre: Skipped 1 previous similar message [ 2223.738479] Lustre: lustre-MDT0001: new disk, initializing [ 2223.769391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2223.780491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2223.783489] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2225.123596] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2228.340152] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2229.589717] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2231.979048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2232.103534] Lustre: lustre-OST0000: new disk, initializing [ 2232.108085] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2232.149471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2234.048085] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2234.870657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2234.873567] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2234.903168] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2237.032008] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2238.952018] Lustre: Failing over lustre-OST0000 [ 2239.023768] Lustre: server umount lustre-OST0000 complete [ 2251.852682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2251.976118] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2251.983230] Lustre: Skipped 1 previous similar message [ 2251.988643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2251.997931] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2253.073558] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2253.868712] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2255.125845] Lustre: Failing over lustre-MDT0000 [ 2255.220651] Lustre: server umount lustre-MDT0000 complete [ 2256.159186] Lustre: 138878:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778349119/real 1778349119] req@000000002db6074c x1864734164936448/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1778349135 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2256.170225] Lustre: 138878:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2257.376723] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2257.381810] Lustre: Skipped 2 previous similar messages [ 2261.471094] Lustre: 138876:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778349124/real 1778349124] req@000000008582199f x1864734164937600/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1778349140 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2262.496086] LustreError: 141443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2262.500651] LustreError: 141443:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2263.044275] LustreError: 140657:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2266.591078] Lustre: 138876:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778349129/real 1778349129] req@00000000cd318e6e x1864734164938752/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1778349145 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2267.616049] LustreError: 140661:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2267.620774] LustreError: 140661:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2267.980692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2268.023338] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2268.026965] Lustre: 140652:0:(mgc_request.c:1917:mgc_process_log()) MGC192.168.204.140@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2268.109266] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2268.127887] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2268.128114] Lustre: lustre-OST0000: Recovery over after 0:15, of 3 clients 3 recovered and 0 were evicted. [ 2268.136159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2268.154796] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2268.164656] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2269.345216] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2273.258950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2273.261081] Lustre: Skipped 1 previous similar message [ 2273.267813] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2273.288733] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2278.371973] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2278.373187] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2278.380335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2278.381981] Lustre: Skipped 1 previous similar message [ 2280.352100] Lustre: server umount lustre-OST0000 complete [ 2283.489847] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2283.490661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2283.496758] Lustre: Skipped 2 previous similar messages [ 2283.497027] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2283.499502] Lustre: Skipped 1 previous similar message [ 2283.501034] LustreError: Skipped 1 previous similar message [ 2288.609143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2288.614265] Lustre: Skipped 1 previous similar message [ 2293.728620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2293.732844] Lustre: Skipped 1 previous similar message [ 2296.287100] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2296.353421] Lustre: server umount lustre-MDT0000 complete [ 2297.732660] LustreError: 141427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778349176 with bad export cookie 8372973436470483592 [ 2297.736725] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2297.738315] LustreError: 141427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2297.862139] Lustre: server umount lustre-MDT0001 complete [ 2301.809166] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 2303.091254] Key type lgssc unregistered [ 2303.266656] LNet: 145881:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2303.271466] LNetError: 145881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2303.282520] LNet: Removed LNI 192.168.204.140@tcp [ 2303.706947] Key type .llcrypt unregistered [ 2303.708164] Key type ._llcrypt unregistered [ 2325.072264] Key type ._llcrypt registered [ 2325.073300] Key type .llcrypt registered [ 2325.110602] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 2325.537583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2325.553323] alg: No test for adler32 (adler32-zlib) [ 2326.406490] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 2326.492195] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 2328.071165] Key type lgssc registered [ 2328.411927] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2331.093619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2331.101602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2332.263952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2333.627413] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2337.192779] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2339.666095] LustreError: 147409:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2339.673687] LustreError: 147409:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2339.682045] LustreError: 147409:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 2339.830192] Lustre: server umount lustre-MDT0000 complete [ 2345.802851] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 2349.353485] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 2361.964731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2362.055301] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2362.065755] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2362.101996] Lustre: lustre-MDT0000: new disk, initializing [ 2362.126761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2362.133475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2363.404750] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2367.927863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2367.965869] Lustre: 150560:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2367.986107] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2367.988056] Lustre: Skipped 1 previous similar message [ 2368.023899] Lustre: lustre-MDT0001: new disk, initializing [ 2368.052257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2368.064278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2368.068700] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2369.386645] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2372.536893] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2373.675467] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2376.048475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2376.156812] Lustre: lustre-OST0000: new disk, initializing [ 2376.158764] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2376.186431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2378.179411] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2379.123562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2379.126489] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2379.150582] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2381.666345] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2384.614347] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2385.191278] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 13:54:23 (1778349263) [ 2449.887570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2449.890201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2449.895697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2450.911916] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2450.912101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2450.920385] Lustre: Skipped 1 previous similar message [ 2450.923076] Lustre: Skipped 1 previous similar message [ 2453.929222] Lustre: server umount lustre-MDT0000 complete [ 2455.236504] LustreError: 150552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778349334 with bad export cookie 12981835746229376777 [ 2455.237614] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2455.242605] LustreError: 150552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2456.031540] LustreError: 150568:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2456.031636] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2456.036921] LustreError: 150568:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2456.046732] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2456.048481] Lustre: Skipped 1 previous similar message [ 2460.063474] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2460.063474] LustreError: 150566:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2461.436206] Lustre: server umount lustre-MDT0001 complete [ 2468.942743] Lustre: server umount lustre-OST0000 complete [ 2472.420915] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 2475.369121] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 2488.903297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2489.051646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2489.072878] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2489.129520] Lustre: lustre-MDT0000: new disk, initializing [ 2489.162394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2489.173676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2490.951406] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2495.906804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2495.949582] Lustre: 156780:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2495.974673] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2495.976720] Lustre: Skipped 1 previous similar message [ 2496.016178] Lustre: lustre-MDT0001: new disk, initializing [ 2496.064728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2496.069698] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2497.449666] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2500.864939] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2502.300313] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2504.906122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2505.047483] Lustre: lustre-OST0000: new disk, initializing [ 2505.051347] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2505.098431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2505.102357] Lustre: Skipped 1 previous similar message [ 2507.152577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2507.160534] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2507.207150] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2507.549869] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2511.011305] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2512.402713] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2512.483286] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2513.837788] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2513.909500] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2516.688855] Lustre: server umount lustre-OST0000 complete [ 2521.568288] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2521.574963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2521.586848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2528.783623] Lustre: server umount lustre-MDT0000 complete [ 2530.419428] LustreError: 159920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778349409 with bad export cookie 12981835746229535593 [ 2530.422138] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2530.427060] LustreError: 159920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2540.091831] Lustre: server umount lustre-MDT0001 complete [ 2543.859316] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 13:57:01 (1778349421) [ 2550.258821] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 2554.460411] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 2566.148599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2566.248075] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2566.259053] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2566.300274] Lustre: lustre-MDT0000: new disk, initializing [ 2566.325459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2566.331975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2567.647642] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2571.911354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2571.954209] Lustre: 163097:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2572.061646] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2573.377442] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2576.532377] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2577.829150] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2580.640600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2582.819861] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2584.519847] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2584.555867] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2585.865412] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2589.665933] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2589.666567] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2589.668417] LustreError: Skipped 1 previous similar message [ 2589.668960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2589.672367] Lustre: Skipped 2 previous similar messages [ 2589.678633] Lustre: Skipped 5 previous similar messages [ 2593.806283] Lustre: server umount lustre-OST0000 complete [ 2597.856282] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2597.862301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2601.264987] Lustre: server umount lustre-MDT0000 complete [ 2602.507720] LustreError: 163090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778349481 with bad export cookie 12981835746229536461 [ 2602.510784] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2602.511347] LustreError: 163090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2602.622043] Lustre: server umount lustre-MDT0001 complete [ 2605.759147] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 13:58:03 (1778349483) [ 2611.129804] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 2614.635201] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 2627.488348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2627.592837] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2627.605404] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2627.608040] Lustre: Skipped 3 previous similar messages [ 2627.651409] Lustre: lustre-MDT0000: new disk, initializing [ 2627.652994] Lustre: Skipped 2 previous similar messages [ 2627.679600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2627.681634] Lustre: Skipped 2 previous similar messages [ 2627.687263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2627.689612] Lustre: Skipped 2 previous similar messages [ 2629.007877] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2633.726160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2633.769185] Lustre: 169008:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2633.868079] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2635.188980] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2638.225309] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2639.431527] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2641.754521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2642.655963] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2643.942319] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2647.147708] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2653.154903] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2653.155701] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2653.157631] LustreError: Skipped 1 previous similar message [ 2653.158935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2653.158969] Lustre: Skipped 3 previous similar messages [ 2653.161555] Lustre: Skipped 2 previous similar messages [ 2655.600522] Lustre: server umount lustre-OST0000 complete [ 2663.388080] Lustre: server umount lustre-MDT0000 complete [ 2663.392356] LustreError: 169019:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2663.397353] LustreError: 169019:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2664.661099] LustreError: 168999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778349543 with bad export cookie 12981835746229537532 [ 2664.664244] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2664.664700] LustreError: 168999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2664.771706] Lustre: server umount lustre-MDT0001 complete [ 2668.873455] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 2669.942156] Key type lgssc unregistered [ 2670.087438] LNet: 172484:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2670.091483] LNetError: 172484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2670.106416] LNet: Removed LNI 192.168.204.140@tcp [ 2670.460995] Key type .llcrypt unregistered [ 2670.462503] Key type ._llcrypt unregistered [ 2682.198209] Key type ._llcrypt registered [ 2682.199246] Key type .llcrypt registered [ 2682.236915] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 2682.597489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2682.626257] alg: No test for adler32 (adler32-zlib) [ 2683.489731] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 2683.569864] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 2685.159244] Key type lgssc registered [ 2685.576667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2688.179447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2688.186153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2689.369772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2690.613323] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2693.426976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2693.564426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2694.837634] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2696.622653] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2697.966313] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2700.935556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2701.102302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2703.177258] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2705.496758] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2714.406100] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2728.928072] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2728.932422] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2728.940492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2729.953196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2729.953351] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2729.954247] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2729.956825] Lustre: Skipped 1 previous similar message [ 2734.029137] Lustre: server umount lustre-OST0000 complete [ 2739.680127] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2739.683642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2739.689687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2741.459228] Lustre: server umount lustre-MDT0000 complete [ 2742.925524] LustreError: 175476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778349621 with bad export cookie 15994809590584622494 [ 2742.928372] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2742.933763] LustreError: 175476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2743.048916] Lustre: server umount lustre-MDT0001 complete [ 2747.093708] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 2748.082817] Key type lgssc unregistered [ 2748.217726] LNet: 177059:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2748.223537] LNetError: 177059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2748.233574] LNet: Removed LNI 192.168.204.140@tcp [ 2748.634176] Key type .llcrypt unregistered [ 2748.635569] Key type ._llcrypt unregistered [ 2758.427935] Key type ._llcrypt registered [ 2758.429864] Key type .llcrypt registered [ 2758.475307] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 2758.842646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2758.869768] alg: No test for adler32 (adler32-zlib) [ 2759.715689] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 2759.797883] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 2761.375181] Key type lgssc registered [ 2761.752809] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2764.669825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2764.680690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2765.866130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2767.217929] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2770.980985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2771.144241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2772.570305] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2774.387247] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2775.558207] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2777.984821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2778.102956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2780.044863] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2782.315088] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2790.375529] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2795.491654] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2795.494724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2795.494777] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2795.501746] Lustre: Skipped 1 previous similar message [ 2798.091085] Lustre: lustre-OST0000: Not available for connect from 192.168.204.40@tcp (stopping) [ 2798.094303] Lustre: Skipped 1 previous similar message [ 2800.611453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2800.615500] Lustre: Skipped 1 previous similar message [ 2801.308462] Lustre: server umount lustre-OST0000 complete [ 2803.535173] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff906906cb3000.ost_server_uuid 50 [ 2804.124507] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2815.968710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2815.969839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2815.975066] Lustre: Skipped 1 previous similar message [ 2820.375137] Lustre: server umount lustre-MDT0000 complete [ 2821.088717] LustreError: 178436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2821.101105] LustreError: 178436:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2821.806033] LustreError: 178416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778349700 with bad export cookie 14654910346641954326 [ 2821.809258] LustreError: 178416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2821.809787] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2821.920880] Lustre: server umount lustre-MDT0001 complete [ 2825.274057] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 14:01:43 (1778349703) [ 2827.955924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2828.122689] LustreError: 181932:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2828.142596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2829.362702] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2832.868648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2833.016840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2834.278593] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2836.191724] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2837.634790] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2840.124128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2842.479712] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2844.377250] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2847.151778] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2847.259173] Lustre: lustre-OST0001: new disk, initializing [ 2847.262417] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2847.305371] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2847.308925] Lustre: Skipped 1 previous similar message [ 2849.403156] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2852.482425] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2852.872163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2852.878886] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2852.916197] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2853.926266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2858.978775] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2858.980478] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2858.981493] LustreError: Skipped 1 previous similar message [ 2858.987145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2858.990600] Lustre: Skipped 2 previous similar messages [ 2860.796367] Lustre: server umount lustre-OST0000 complete [ 2862.861967] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2862.934410] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2863.115316] LustreError: 183526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2864.168645] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2864.232428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2868.229419] LustreError: 185038:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2868.234814] LustreError: 185038:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2873.349952] LustreError: 183525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2873.357792] LustreError: 183525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2874.342317] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2874.349276] Lustre: Skipped 1 previous similar message [ 2874.352753] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2874.356712] Lustre: Skipped 1 previous similar message [ 2880.427667] Lustre: server umount lustre-OST0001 complete [ 2884.577146] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2884.577363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2884.587875] Lustre: Skipped 2 previous similar messages [ 2894.816206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2894.818638] Lustre: Skipped 7 previous similar messages [ 2896.351086] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2896.418579] Lustre: server umount lustre-MDT0000 complete [ 2897.654584] LustreError: 184435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778349776 with bad export cookie 14654910346641955383 [ 2897.658596] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2897.660530] LustreError: 184435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2897.796144] Lustre: server umount lustre-MDT0001 complete [ 2905.979786] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 14:03:04 (1778349784) [ 2909.057885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2909.159975] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2909.172164] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2909.248951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2910.642072] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2913.785825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2913.834095] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2913.859207] Lustre: 187704:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2915.441852] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2917.787193] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2919.321569] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2921.941700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2922.049062] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2924.535478] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2927.104530] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2929.192171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 2930.104050] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2930.148418] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2932.673353] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2934.774083] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2935.333085] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 2939.359986] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2939.360201] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2939.362690] LustreError: Skipped 1 previous similar message [ 2939.366513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2939.369834] Lustre: Skipped 1 previous similar message [ 2950.111202] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2950.143893] Lustre: server umount lustre-OST0000 complete [ 2950.627304] LustreError: 190790:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2950.632629] LustreError: 190790:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2960.866093] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2960.866184] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2960.870835] Lustre: Skipped 1 previous similar message [ 2967.004543] Lustre: server umount lustre-OST0001 complete [ 2970.591766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2970.593925] LustreError: Skipped 1 previous similar message [ 2974.647949] Lustre: server umount lustre-MDT0000 complete [ 2976.069169] LustreError: 187741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778349855 with bad export cookie 14654910346641956769 [ 2976.073428] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2976.073556] LustreError: 187741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2976.201129] Lustre: server umount lustre-MDT0001 complete [ 2984.498313] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 14:04:22 (1778349862) [ 2989.867824] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 2993.265451] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 3007.225827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3007.325459] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3007.335983] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3007.375492] Lustre: lustre-MDT0000: new disk, initializing [ 3007.400289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3007.403104] Lustre: Skipped 3 previous similar messages [ 3007.409081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3008.742617] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3013.050367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3013.085854] Lustre: 195277:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3013.105574] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3013.107303] Lustre: Skipped 1 previous similar message [ 3013.149031] Lustre: lustre-MDT0001: new disk, initializing [ 3013.190898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3013.194274] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3014.447741] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3017.682159] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3018.935961] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3021.423153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3021.513671] Lustre: lustre-OST0000: new disk, initializing [ 3021.516937] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3021.909449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3021.912607] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3021.946177] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3023.541265] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3026.758982] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3028.149943] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3028.218680] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3029.591182] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3029.683878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3032.545476] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3032.547058] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3032.550275] LustreError: Skipped 1 previous similar message [ 3032.556630] Lustre: Skipped 3 previous similar messages [ 3032.557915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3032.564775] Lustre: Skipped 14 previous similar messages [ 3036.592165] Lustre: server umount lustre-OST0000 complete [ 3044.317229] Lustre: server umount lustre-MDT0000 complete [ 3045.776298] LustreError: 195269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778349924 with bad export cookie 14654910346641958540 [ 3045.779392] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3045.782215] LustreError: 195269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3045.903710] Lustre: server umount lustre-MDT0001 complete [ 3049.172717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3049.327957] LustreError: 199086:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3049.334139] LustreError: 199086:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 3050.625610] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3054.163284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3055.697574] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3057.579720] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3058.848031] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3061.794227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3064.194451] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3066.726516] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3068.488393] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3074.772399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec [ 3076.132109] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3076.214548] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3079.650719] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3079.652848] LustreError: Skipped 2 previous similar messages [ 3083.106870] Lustre: server umount lustre-OST0000 complete [ 3085.349935] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3086.718334] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3110.581389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3110.717209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3110.719938] Lustre: Skipped 5 previous similar messages [ 3112.902952] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3115.346765] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3115.943817] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3117.212274] Lustre: server umount lustre-OST0000 complete [ 3119.713479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3119.719236] Lustre: Skipped 6 previous similar messages [ 3124.842382] Lustre: server umount lustre-MDT0000 complete [ 3125.217241] LustreError: 201549:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3125.225098] LustreError: 201549:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3126.156233] LustreError: 199112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778350005 with bad export cookie 14654910346641959408 [ 3126.159366] LustreError: 199112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3126.159721] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3126.276977] Lustre: server umount lustre-MDT0001 complete [ 3128.991950] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 14:06:47 (1778350007) [ 3131.796476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3133.300303] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3136.086869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3137.595833] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3139.525042] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3140.699434] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3143.315101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3145.482814] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3147.441809] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3148.947269] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3158.242903] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 3159.470821] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3159.559343] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3162.228797] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3162.324269] Lustre: lustre-OST0001: new disk, initializing [ 3162.326757] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3164.256964] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3167.824489] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3169.274362] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3170.307739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3170.311872] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3170.342623] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3171.421889] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3172.574870] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3172.642628] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3175.398476] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3175.401287] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3175.402713] LustreError: Skipped 1 previous similar message [ 3175.406875] Lustre: Skipped 8 previous similar messages [ 3179.406290] Lustre: server umount lustre-OST0001 complete [ 3181.753282] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3181.822215] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3183.361496] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3183.455858] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3190.758263] LustreError: 208992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3190.769266] LustreError: 208992:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 3208.019487] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3210.187740] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3212.688862] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3213.291328] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3213.297774] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3213.297897] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3213.307047] Lustre: Skipped 1 previous similar message [ 3213.453474] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3220.479637] Lustre: server umount lustre-OST0001 complete [ 3239.180550] Lustre: server umount lustre-OST0000 complete [ 3246.672459] Lustre: server umount lustre-MDT0000 complete [ 3247.955619] LustreError: 207632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778350127 with bad export cookie 14654910346641960339 [ 3247.962352] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3247.962810] LustreError: 207632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3254.125182] Lustre: server umount lustre-MDT0001 complete [ 3262.689479] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 14:09:00 (1778350140) [ 3264.873139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3264.962911] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3264.973443] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3265.053305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3265.056518] Lustre: Skipped 5 previous similar messages [ 3266.352129] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3270.077850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3270.127877] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3270.144111] Lustre: 212588:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3271.685910] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3273.561955] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3275.181369] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3277.823519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3277.894546] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3279.904551] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3282.260620] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3291.463317] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3291.500559] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3293.641232] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3295.924930] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3296.500649] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3298.002863] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3298.107956] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3299.590155] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3299.651323] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3300.842710] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff906907b1a800.ost_server_uuid 50 [ 3301.496612] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff906907b1a800.ost_server_uuid in FULL state after 0 sec [ 3301.958201] Lustre: Permanently deactivating lustre-OST0001 [ 3301.965572] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3302.776857] Lustre: Permanently reactivating lustre-OST0001 [ 3306.979203] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3306.979760] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3306.983296] Lustre: Skipped 8 previous similar messages [ 3306.987011] LustreError: Skipped 5 previous similar messages [ 3309.891519] Lustre: server umount lustre-OST0001 complete [ 3317.394647] Lustre: server umount lustre-OST0000 complete [ 3324.700101] Lustre: server umount lustre-MDT0000 complete [ 3326.191872] LustreError: 213313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778350205 with bad export cookie 14654910346641961613 [ 3326.198198] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3326.199820] LustreError: 213313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3326.353092] Lustre: server umount lustre-MDT0001 complete [ 3335.463221] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 14:10:13 (1778350213) [ 3338.353485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3338.470137] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3338.486724] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3338.496284] Lustre: Skipped 5 previous similar messages [ 3340.028977] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3342.927585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3342.983516] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3342.998840] Lustre: 218691:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3344.439534] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3346.079337] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3347.250931] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3349.567622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3349.645479] Lustre: Permanently deactivating lustre-OST0000 [ 3349.649613] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3351.634400] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3353.308838] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3355.686355] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3357.694608] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3357.736059] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3357.909502] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3359.816511] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3360.343718] Lustre: Permanently reactivating lustre-OST0000 [ 3366.899496] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3366.901887] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3366.905101] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3366.919608] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3366.925618] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3376.456340] Lustre: server umount lustre-OST0001 complete [ 3378.148687] LustreError: 220310:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3378.154537] LustreError: 220310:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 22 previous similar messages [ 3384.267063] Lustre: server umount lustre-OST0000 complete [ 3391.656152] Lustre: server umount lustre-MDT0000 complete [ 3393.221983] LustreError: 218726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778350272 with bad export cookie 14654910346641963251 [ 3393.225112] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3393.225438] LustreError: 218726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3393.343444] Lustre: server umount lustre-MDT0001 complete [ 3397.235385] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 3398.676270] Key type lgssc unregistered [ 3398.851702] LNet: 223066:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3398.855807] LNetError: 223066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3398.870546] LNet: Removed LNI 192.168.204.140@tcp [ 3399.287853] Key type .llcrypt unregistered [ 3399.288946] Key type ._llcrypt unregistered [ 3409.200103] Key type ._llcrypt registered [ 3409.201148] Key type .llcrypt registered [ 3409.231303] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 3409.565963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3409.588725] alg: No test for adler32 (adler32-zlib) [ 3410.461612] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 3410.566198] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 3412.167246] Key type lgssc registered [ 3412.637519] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3416.457553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3416.473478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3417.720771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3419.214638] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3422.277373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3422.319162] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3422.328511] Lustre: 224483:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3422.332444] Lustre: Permanently deactivating lustre-MDT0001 [ 3422.337323] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3422.437894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3423.806765] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3424.495604] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3424.499444] LustreError: 224523:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3424.505232] LustreError: 224523:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3424.509335] LustreError: 224523:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 1, failed: rc = -108 [ 3425.804471] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3427.243751] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3430.117103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3430.269796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3432.359955] LustreError: 226394:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3432.362494] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3432.370175] LustreError: 226394:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3432.603724] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3435.332726] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3437.542125] LustreError: 226394:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3438.530944] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3438.627958] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3439.880337] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3440.995464] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3441.970604] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3441.974739] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3443.638895] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3445.336735] Lustre: Permanently reactivating lustre-MDT0001 [ 3445.343760] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3445.346485] Lustre: Skipped 2 previous similar messages [ 3449.870767] Lustre: lustre-MDT0001: Client a274ffce-9f82-4ef2-b20b-764c85fac68d (at 192.168.204.40@tcp) reconnecting [ 3451.887320] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3452.412734] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3454.575740] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3454.667481] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3456.307060] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3456.410881] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3457.578331] Lustre: Permanently deactivating lustre-MDT0001 [ 3457.586163] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3457.589675] Lustre: Skipped 2 previous similar messages [ 3466.779300] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3473.889138] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3473.894433] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3473.903201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3474.914088] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3474.914391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3474.918064] Lustre: Skipped 2 previous similar messages [ 3474.920861] Lustre: Skipped 3 previous similar messages [ 3476.209776] Lustre: server umount lustre-MDT0000 complete [ 3477.960217] LustreError: 225214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778350357 with bad export cookie 7495883346097713706 [ 3477.965617] LustreError: 225214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3477.966269] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3478.133368] Lustre: server umount lustre-MDT0001 complete [ 3485.835212] Lustre: server umount lustre-OST0000 complete [ 3490.560564] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 14:12:48 (1778350368) [ 3494.018413] Lustre: server umount lustre-OST0001 complete [ 3496.405294] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 3500.247898] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 3513.084356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3513.169241] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3513.173574] Lustre: Skipped 2 previous similar messages [ 3513.182422] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3513.217687] Lustre: lustre-MDT0000: new disk, initializing [ 3513.240845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3513.250764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3514.521633] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3518.611782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3518.649309] Lustre: 232188:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3518.671610] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3518.673348] Lustre: Skipped 1 previous similar message [ 3518.709304] Lustre: lustre-MDT0001: new disk, initializing [ 3518.755039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3518.757740] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3520.051115] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3523.167393] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3524.294870] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3526.549073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3526.656843] Lustre: lustre-OST0000: new disk, initializing [ 3526.658860] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3526.690629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3526.692902] Lustre: Skipped 1 previous similar message [ 3527.130194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3527.134154] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3527.169385] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3528.511413] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3531.598651] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3532.266371] LustreError: 234577:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3536.687402] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3536.756548] Lustre: lustre-OST0001: new disk, initializing [ 3536.758473] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3538.656385] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3541.776519] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3542.346650] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3542.359067] LustreError: 234577:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3542.368051] LustreError: 235525:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3544.058630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3544.063446] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3544.088863] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3552.464062] LustreError: 235525:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3554.280070] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3554.281748] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3554.284661] LustreError: Skipped 1 previous similar message [ 3554.295443] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3554.302568] Lustre: Skipped 1 previous similar message [ 3559.395119] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3559.397545] Lustre: Skipped 1 previous similar message [ 3559.689356] Lustre: server umount lustre-OST0001 complete [ 3564.520428] LustreError: 234017:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3564.531577] LustreError: 234017:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3569.637634] LustreError: 233916:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3569.645535] LustreError: 233916:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3573.217172] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3573.218468] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3573.223139] LustreError: Skipped 1 previous similar message [ 3573.224288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3573.233270] Lustre: Skipped 2 previous similar messages [ 3574.756349] LustreError: 233914:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3574.762834] LustreError: 233914:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3578.160581] Lustre: server umount lustre-OST0000 complete [ 3579.872288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3579.875922] Lustre: Skipped 1 previous similar message [ 3584.992113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3584.994928] Lustre: Skipped 6 previous similar messages [ 3594.207164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3594.277938] Lustre: server umount lustre-MDT0000 complete [ 3595.231904] LustreError: 232199:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3595.238488] LustreError: 232199:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3595.592165] LustreError: 232180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778350474 with bad export cookie 7495883346097716485 [ 3595.595501] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3595.596134] LustreError: 232180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3601.799716] Lustre: server umount lustre-MDT0001 complete [ 3605.706828] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 3606.675861] Key type lgssc unregistered [ 3606.805358] LNet: 236852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3606.807674] LNetError: 236852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3606.813366] LNet: Removed LNI 192.168.204.140@tcp [ 3607.111828] Key type .llcrypt unregistered [ 3607.113356] Key type ._llcrypt unregistered [ 3622.442767] Key type ._llcrypt registered [ 3622.444370] Key type .llcrypt registered [ 3622.479742] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 3622.890271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3622.944711] alg: No test for adler32 (adler32-zlib) [ 3623.806484] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 3623.898073] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 3625.487135] Key type lgssc registered [ 3626.049239] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3629.707162] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3629.716388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3630.821920] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3630.835866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3630.906553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3632.292995] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3635.877416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3635.914919] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3635.925389] Lustre: 239019:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3636.018359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3637.389900] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3639.518895] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3640.911720] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3644.166932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3644.289525] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3644.402630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3646.907055] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3648.209217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 3649.102594] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3689.442752] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3689.446163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3689.448727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3689.455271] Lustre: Skipped 1 previous similar message [ 3694.511355] Lustre: server umount lustre-OST0000 complete [ 3696.442616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3701.794169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3701.943722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3704.055202] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3706.108188] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3709.097748] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3709.104738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3709.104993] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3709.113032] Lustre: Skipped 1 previous similar message [ 3740.128429] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3740.130711] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3740.131725] LustreError: Skipped 2 previous similar messages [ 3740.135903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3740.139914] Lustre: Skipped 2 previous similar messages [ 3745.250415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3745.254140] Lustre: Skipped 1 previous similar message [ 3745.523349] Lustre: server umount lustre-OST0000 complete [ 3748.832343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3748.838258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3748.847601] Lustre: Skipped 1 previous similar message [ 3748.851387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3753.393928] Lustre: server umount lustre-MDT0000 complete [ 3755.003767] LustreError: 239010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778350634 with bad export cookie 16408377824105022601 [ 3755.009131] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3755.015492] LustreError: 239010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3755.155386] Lustre: server umount lustre-MDT0001 complete [ 3758.851320] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 3759.829581] Key type lgssc unregistered [ 3759.953488] LNet: 243966:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3759.956203] LNetError: 243966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3759.965329] LNet: Removed LNI 192.168.204.140@tcp [ 3760.341992] Key type .llcrypt unregistered [ 3760.343564] Key type ._llcrypt unregistered [ 3769.650639] Key type ._llcrypt registered [ 3769.651809] Key type .llcrypt registered [ 3769.692441] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 3770.039178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3770.051386] alg: No test for adler32 (adler32-zlib) [ 3770.929238] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 3771.041281] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 3772.631237] Key type lgssc registered [ 3773.175458] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3776.264638] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3776.280798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3777.492595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3779.082602] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3782.372664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3782.532390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3784.172699] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3786.389301] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3787.845181] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3790.571732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3790.754842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3793.222962] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3795.308531] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3802.030884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3817.442275] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3817.443301] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3817.448243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3817.452413] Lustre: Skipped 1 previous similar message [ 3821.209991] Lustre: server umount lustre-OST0000 complete [ 3823.584400] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3823.586566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3823.590466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3823.592094] Lustre: Skipped 1 previous similar message [ 3827.681722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3827.683344] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3828.992596] Lustre: server umount lustre-MDT0000 complete [ 3830.578619] LustreError: 245325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778350709 with bad export cookie 7322665149433610944 [ 3830.584277] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3830.588297] LustreError: 245325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3830.726425] Lustre: server umount lustre-MDT0001 complete [ 3834.899910] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 3836.253291] Key type lgssc unregistered [ 3836.408458] LNet: 249318:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3836.410794] LNetError: 249318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3836.418464] LNet: Removed LNI 192.168.204.140@tcp [ 3836.828495] Key type .llcrypt unregistered [ 3836.830026] Key type ._llcrypt unregistered [ 3844.638332] Key type ._llcrypt registered [ 3844.639486] Key type .llcrypt registered [ 3844.680685] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 3844.995670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3845.003112] alg: No test for adler32 (adler32-zlib) [ 3845.872330] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 3845.966054] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 3847.559208] Key type lgssc registered [ 3848.001991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3850.951675] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3850.958489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3852.145336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3853.647466] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3857.233119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3857.381449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3858.793249] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3860.677926] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3862.092644] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3864.877865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3865.052268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3867.107908] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3869.375940] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3877.352469] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3887.588721] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3887.589954] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3887.599376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3887.601699] Lustre: Skipped 1 previous similar message [ 3890.015899] Lustre: server umount lustre-OST0000 complete [ 3892.704553] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3892.708437] Lustre: Skipped 1 previous similar message [ 3892.710248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3892.712319] Lustre: Skipped 1 previous similar message [ 3897.699646] Lustre: server umount lustre-MDT0000 complete [ 3897.823995] LustreError: 250499:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3897.831160] LustreError: 250499:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3899.122080] LustreError: 250480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778350778 with bad export cookie 108060831343195487 [ 3899.125500] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3899.126253] LustreError: 250480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3899.248113] Lustre: server umount lustre-MDT0001 complete [ 3903.682305] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 3904.952837] Key type lgssc unregistered [ 3905.087591] LNet: 253843:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3905.091508] LNetError: 253843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3905.101370] LNet: Removed LNI 192.168.204.140@tcp [ 3905.489179] Key type .llcrypt unregistered [ 3905.491520] Key type ._llcrypt unregistered [ 3915.480612] Key type ._llcrypt registered [ 3915.481728] Key type .llcrypt registered [ 3915.536004] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 3915.952282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3915.999081] alg: No test for adler32 (adler32-zlib) [ 3916.865392] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 3916.956699] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 3918.567225] Key type lgssc registered [ 3919.027114] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3922.122925] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3922.129191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3923.295815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3924.513215] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3927.177145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3927.306576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3928.571833] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3930.706298] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3932.135695] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3934.640855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3934.772415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3936.958604] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3939.259542] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3948.071583] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3960.684921] Lustre: server umount lustre-OST0000 complete [ 3963.362483] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3963.364458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3963.366267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3963.380041] Lustre: Skipped 1 previous similar message [ 3968.480672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3968.483442] Lustre: Skipped 2 previous similar messages [ 3972.622885] Lustre: server umount lustre-MDT0000 complete [ 3973.600986] LustreError: 257603:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3973.610601] LustreError: 257603:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3974.469920] LustreError: 255934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778350853 with bad export cookie 8592576646032081881 [ 3974.475672] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3974.478186] LustreError: 255934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3984.945844] Lustre: server umount lustre-MDT0001 complete [ 3989.474470] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 3991.005388] Key type lgssc unregistered [ 3991.187691] LNet: 259188:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3991.193405] LNetError: 259188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3991.207527] LNet: Removed LNI 192.168.204.140@tcp [ 3991.684966] Key type .llcrypt unregistered [ 3991.687518] Key type ._llcrypt unregistered [ 4000.113082] Key type ._llcrypt registered [ 4000.114362] Key type .llcrypt registered [ 4000.151796] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 4000.414714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4000.437216] alg: No test for adler32 (adler32-zlib) [ 4001.284102] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 4001.368071] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 4002.951187] Key type lgssc registered [ 4003.568156] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4007.227099] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4007.239658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4008.447644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4009.852316] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4013.096654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4013.256441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4014.685370] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4016.349135] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4017.424298] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4019.674277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4019.803836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4021.761562] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4023.546092] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4033.126323] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4038.626623] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4038.626726] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4038.627454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4038.630558] Lustre: Skipped 1 previous similar message [ 4039.501490] Lustre: server umount lustre-OST0000 complete [ 4042.838847] Lustre: server umount lustre-MDT0000 complete [ 4043.744863] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4043.749525] LustreError: 260379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4043.755226] LustreError: 260379:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4044.188154] LustreError: 260350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778350923 with bad export cookie 5040936676930263443 [ 4044.191040] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4044.192705] LustreError: 260350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4044.312491] Lustre: server umount lustre-MDT0001 complete [ 4047.499760] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 4048.389996] Key type lgssc unregistered [ 4048.509336] LNet: 263748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4048.511699] LNetError: 263748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4048.522270] LNet: Removed LNI 192.168.204.140@tcp [ 4048.781750] Key type .llcrypt unregistered [ 4048.783601] Key type ._llcrypt unregistered [ 4052.243647] Key type ._llcrypt registered [ 4052.244772] Key type .llcrypt registered [ 4052.285381] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4057.540469] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 4065.130199] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 4065.552906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4065.564405] alg: No test for adler32 (adler32-zlib) [ 4066.427453] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 4066.507585] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 4068.087183] Key type lgssc registered [ 4068.514200] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4082.678301] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4082.687179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4083.798781] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4083.811313] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4083.850499] Lustre: lustre-MDT0000: new disk, initializing [ 4083.877843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4083.885193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4085.369667] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4090.329964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4090.387809] Lustre: 267221:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4090.424537] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4090.428075] Lustre: Skipped 1 previous similar message [ 4090.471441] Lustre: lustre-MDT0001: new disk, initializing [ 4090.514702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4090.539266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4090.545582] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4091.942272] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4095.736318] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4097.291255] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4100.542717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4100.693123] Lustre: lustre-OST0000: new disk, initializing [ 4100.696092] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4100.729995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4103.311705] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4107.150431] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4108.799773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4108.805540] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4108.832739] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4108.841967] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4108.907835] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4110.410174] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4110.473370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4113.890336] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4113.893349] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4113.895794] LustreError: Skipped 1 previous similar message [ 4113.896549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4113.902709] Lustre: Skipped 1 previous similar message [ 4113.913606] Lustre: Skipped 1 previous similar message [ 4117.263859] Lustre: server umount lustre-OST0000 complete [ 4119.009312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4119.010383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4119.012672] Lustre: Skipped 1 previous similar message [ 4119.016350] Lustre: Skipped 1 previous similar message [ 4124.128963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4124.133121] Lustre: Skipped 1 previous similar message [ 4129.248994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4129.253236] Lustre: Skipped 1 previous similar message [ 4133.343165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4133.424783] Lustre: server umount lustre-MDT0000 complete [ 4135.063163] LustreError: 267214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778351014 with bad export cookie 18265817650855580854 [ 4135.066278] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4135.066950] LustreError: 267214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4135.202419] Lustre: server umount lustre-MDT0001 complete [ 4138.241288] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 14:23:36 (1778351016) [ 4141.132250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4141.299558] LustreError: 271228:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4141.304955] LustreError: 271228:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4141.324238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4142.628046] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4145.674899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4145.798801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4146.950559] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4148.539224] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4149.998707] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4152.731801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4155.713084] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4158.277620] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4172.775293] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4172.777796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4172.782639] Lustre: Skipped 1 previous similar message [ 4172.783105] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4172.786436] Lustre: Skipped 1 previous similar message [ 4177.892129] Lustre: server umount lustre-OST0000 complete [ 4181.984581] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4181.986766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4181.989536] LustreError: Skipped 1 previous similar message [ 4181.999686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4185.950932] Lustre: server umount lustre-MDT0000 complete [ 4187.327063] LustreError: 271209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778351066 with bad export cookie 18265817650855581722 [ 4187.331613] LustreError: 271948:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4187.332871] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4187.334091] LustreError: 271209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4187.479793] Lustre: server umount lustre-MDT0001 complete [ 4191.879989] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 4193.052299] Key type lgssc unregistered [ 4193.171523] LNet: 274426:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4193.173870] LNetError: 274426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4193.183368] LNet: Removed LNI 192.168.204.140@tcp [ 4193.474668] Key type .llcrypt unregistered [ 4193.476741] Key type ._llcrypt unregistered [ 4205.528541] Key type ._llcrypt registered [ 4205.529827] Key type .llcrypt registered [ 4205.591397] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 4212.597231] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 4213.003591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4213.050947] alg: No test for adler32 (adler32-zlib) [ 4213.920167] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 4214.016380] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 4215.615140] Key type lgssc registered [ 4215.963983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4236.292692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4236.302576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4237.414856] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4237.429718] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4237.470150] Lustre: lustre-MDT0000: new disk, initializing [ 4237.497324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4237.505447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4238.855423] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4244.492184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4244.565069] Lustre: 278189:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4244.605625] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4244.608881] Lustre: Skipped 1 previous similar message [ 4244.650716] Lustre: lustre-MDT0001: new disk, initializing [ 4244.707201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4244.727855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4244.734489] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4246.289643] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4250.146217] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4251.672867] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4254.611836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4254.756204] Lustre: lustre-OST2710: new disk, initializing [ 4254.760205] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4254.803492] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4257.137422] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4260.801766] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4262.925609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4262.932890] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4262.970319] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4263.697437] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4263.802871] Lustre: lustre-OST03e8: new disk, initializing [ 4263.807176] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4263.861403] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4266.533695] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4269.073286] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4272.375992] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4272.498770] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4272.659198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4272.663507] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4272.698644] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4274.419537] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4274.531169] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4276.200402] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4276.328289] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4278.102390] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4278.226269] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4280.799746] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4280.802655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4280.809633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4282.850043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4282.851146] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4282.861519] Lustre: Skipped 2 previous similar messages [ 4285.725296] Lustre: server umount lustre-MDT0000 complete [ 4286.891158] LustreError: 280879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778351165 with bad export cookie 11675064231369973796 [ 4286.894041] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4286.896093] LustreError: 280879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4287.003369] Lustre: server umount lustre-MDT0001 complete [ 4298.771525] Lustre: server umount lustre-OST2710 complete [ 4310.881369] Lustre: server umount lustre-OST03e8 complete [ 4318.273663] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 4322.375218] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 4337.856883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4337.945905] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4337.959532] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4338.003460] Lustre: lustre-MDT0000: new disk, initializing [ 4338.035673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4338.044462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4339.688766] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4344.883898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4344.939733] Lustre: 285905:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4345.073929] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4346.707897] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4350.425301] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4351.879070] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4355.089380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4355.247912] Lustre: lustre-OST0000: new disk, initializing [ 4355.251038] Lustre: Skipped 1 previous similar message [ 4355.254701] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4355.259772] Lustre: Skipped 2 previous similar messages [ 4355.308310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4355.312968] Lustre: Skipped 1 previous similar message [ 4357.608026] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4361.224335] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4361.743858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4361.749654] Lustre: Skipped 1 previous similar message [ 4361.753263] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4361.795319] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4362.916175] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4363.021388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4364.532022] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4364.621839] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4366.820172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4366.820225] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4366.822934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4366.822943] Lustre: Skipped 3 previous similar messages [ 4366.825232] LustreError: Skipped 1 previous similar message [ 4366.839235] Lustre: Skipped 1 previous similar message [ 4371.693724] Lustre: server umount lustre-OST0000 complete [ 4376.032443] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4376.037069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4376.045416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4376.049097] Lustre: Skipped 1 previous similar message [ 4379.383563] Lustre: server umount lustre-MDT0000 complete [ 4380.924299] LustreError: 285896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778351259 with bad export cookie 11675064231369975455 [ 4380.929881] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4380.934684] LustreError: 285896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4381.078708] Lustre: server umount lustre-MDT0001 complete [ 4384.862869] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 14:27:42 (1778351262) [ 4385.590962] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4386.429122] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 14:27:44 (1778351264) [ 4387.736820] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 4399.861559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4399.978490] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4399.993709] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4400.086806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4401.705436] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4405.586611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4405.646471] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4405.659721] Lustre: 291333:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4407.792873] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4410.378070] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4412.048463] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4428.452362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4428.561900] Lustre: Denying initial registration attempt from nid 192.168.204.140@tcp, specified as failover [ 4428.567872] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4428.574091] LustreError: 292947:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -99 [ 4428.578789] LustreError: 292947:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 4428.583079] LustreError: 292947:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 4428.620457] Lustre: server umount lustre-OST0000 complete [ 4428.623474] LustreError: 292947:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4435.425749] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4435.427197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4435.435225] Lustre: Skipped 2 previous similar messages [ 4435.439357] Lustre: Skipped 2 previous similar messages [ 4437.554341] Lustre: server umount lustre-MDT0000 complete [ 4439.282620] LustreError: 292057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778351318 with bad export cookie 11675064231369976323 [ 4439.288384] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4439.290540] LustreError: 292057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4439.426322] Lustre: server umount lustre-MDT0001 complete [ 4443.608619] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 4444.755289] Key type lgssc unregistered [ 4444.921612] LNet: 294050:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4444.926750] LNetError: 294050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4444.937605] LNet: Removed LNI 192.168.204.140@tcp [ 4445.391368] Key type .llcrypt unregistered [ 4445.393695] Key type ._llcrypt unregistered [ 4449.786172] Key type ._llcrypt registered [ 4449.787428] Key type .llcrypt registered [ 4449.821413] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 4450.141965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4450.178309] alg: No test for adler32 (adler32-zlib) [ 4451.014787] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 4451.088049] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 4452.663173] Key type lgssc registered [ 4453.147465] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4490.637731] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 4493.583908] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4493.589643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4494.674115] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4494.683030] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4494.740261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4495.922420] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4498.819407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4498.867112] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4498.877680] Lustre: 296763:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4498.976219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4500.314048] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4502.023877] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4503.121500] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4518.498326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4518.570465] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4518.631542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4520.315866] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4521.922232] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4522.592509] Lustre: server umount lustre-OST0000 complete [ 4524.000277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4524.000703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4524.003922] Lustre: Skipped 1 previous similar message [ 4525.038483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4525.040553] Lustre: Skipped 2 previous similar messages [ 4530.144440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4530.147431] Lustre: Skipped 1 previous similar message [ 4535.266060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4535.270947] Lustre: Skipped 1 previous similar message [ 4538.335154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4538.437317] Lustre: server umount lustre-MDT0000 complete [ 4540.320985] LustreError: 296753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778351419 with bad export cookie 9369095856267128144 [ 4540.327348] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4540.328752] LustreError: 296753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4540.474775] Lustre: server umount lustre-MDT0001 complete [ 4544.931085] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 4546.402568] Key type lgssc unregistered [ 4546.569499] LNet: 299968:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4546.574391] LNetError: 299968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4546.586579] LNet: Removed LNI 192.168.204.140@tcp [ 4547.059700] Key type .llcrypt unregistered [ 4547.061249] Key type ._llcrypt unregistered [ 4557.562885] Key type ._llcrypt registered [ 4557.564689] Key type .llcrypt registered [ 4557.625385] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 4557.976355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4558.004068] alg: No test for adler32 (adler32-zlib) [ 4558.850108] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 4558.934847] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 4560.511155] Key type lgssc registered [ 4561.071916] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4564.679260] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4564.690443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4565.897593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4567.517807] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4571.663239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4571.856200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4573.573954] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4576.059185] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4577.888618] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4583.077308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4583.292188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4585.788862] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4588.247991] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4602.847745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4602.851549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4602.858053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4604.950049] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.40@tcp (stopping) [ 4606.947963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4606.949108] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4609.008668] Lustre: server umount lustre-MDT0000 complete [ 4610.053628] LustreError: 301342:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4610.061881] LustreError: 301342:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4610.528346] LustreError: 302963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778351489 with bad export cookie 14370514892671867635 [ 4610.531927] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4610.533165] LustreError: 302963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4612.064601] LustreError: 301343:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4612.064976] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4612.084441] Lustre: Skipped 1 previous similar message [ 4612.088479] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4612.092240] Lustre: Skipped 2 previous similar messages [ 4615.071913] LustreError: 301371:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4616.733995] Lustre: server umount lustre-MDT0001 complete [ 4619.946854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4624.008555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4635.616059] LustreError: 300657:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c255647b x1864736636429440/t0(0) o250->MGC192.168.204.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4635.797368] LustreError: 302951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4635.808085] LustreError: 302951:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4635.842919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4635.889391] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4637.478669] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4639.969080] LustreError: 304623:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4639.978746] LustreError: 304623:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4641.588666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4641.722564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4643.269460] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4645.499886] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4645.800210] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4651.442967] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 4653.161584] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4655.994710] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 4661.219420] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4661.224516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4661.228064] Lustre: Skipped 1 previous similar message [ 4661.237043] Lustre: Skipped 3 previous similar messages [ 4663.793984] Lustre: server umount lustre-OST0000 complete [ 4666.337776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4666.346068] Lustre: Skipped 1 previous similar message [ 4671.457045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4671.462083] Lustre: Skipped 3 previous similar messages [ 4671.665974] Lustre: server umount lustre-MDT0000 complete [ 4673.470248] LustreError: 301326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778351552 with bad export cookie 14370514892671874789 [ 4673.475470] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4673.478972] LustreError: 301326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4673.628671] Lustre: server umount lustre-MDT0001 complete [ 4677.720451] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 4679.254627] Key type lgssc unregistered [ 4679.421538] LNet: 307055:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4679.427370] LNetError: 307055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4679.436559] LNet: Removed LNI 192.168.204.140@tcp [ 4679.893775] Key type .llcrypt unregistered [ 4679.896834] Key type ._llcrypt unregistered [ 4691.099779] Key type ._llcrypt registered [ 4691.102158] Key type .llcrypt registered [ 4691.170672] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 4691.705597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4691.713421] alg: No test for adler32 (adler32-zlib) [ 4692.560702] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 4692.643218] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 4694.215200] Key type lgssc registered [ 4694.776283] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4698.432236] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4698.445731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4699.620146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4701.367538] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4705.374527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4705.558216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4707.343592] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4709.494372] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4710.950662] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4714.977674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4714.978694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4714.985977] Lustre: Skipped 1 previous similar message [ 4714.993013] Lustre: Skipped 1 previous similar message [ 4716.001478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4716.007184] Lustre: Skipped 1 previous similar message [ 4718.652267] Lustre: server umount lustre-MDT0000 complete [ 4719.008991] LustreError: 308431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4719.019124] LustreError: 308431:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4720.432992] LustreError: 308413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778351599 with bad export cookie 14064210117397776091 [ 4720.437176] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4720.551804] Lustre: server umount lustre-MDT0001 complete [ 4726.832489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4726.954341] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4726.970156] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4727.083947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4728.769415] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4732.984123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4733.029651] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4733.042320] Lustre: 310790:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4733.182469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4734.887885] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4737.070728] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4738.319656] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4743.655569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4743.767757] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4743.774769] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4743.780058] LustreError: 312446:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -2 [ 4743.783908] LustreError: 312446:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 4743.787111] LustreError: 312446:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 4743.820459] Lustre: server umount lustre-OST0000 complete [ 4743.823665] LustreError: 312446:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4749.987362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4750.077057] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4750.179160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4752.061026] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4753.896503] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4757.451537] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4757.507555] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4757.552108] Lustre: lustre-OST0001: new disk, initializing [ 4757.556098] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4758.135109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4758.140170] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4758.174625] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 4758.211225] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4760.082556] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4762.450935] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4768.737140] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4768.739823] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4768.741930] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4768.743263] Lustre: Skipped 1 previous similar message [ 4768.750024] Lustre: Skipped 1 previous similar message [ 4769.858058] Lustre: server umount lustre-OST0001 complete [ 4773.860678] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4773.861068] LustreError: 312912:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4773.861280] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4773.861302] LustreError: Skipped 1 previous similar message [ 4773.863663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4773.868671] Lustre: Skipped 1 previous similar message [ 4773.893333] LustreError: 312912:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4777.712691] Lustre: server umount lustre-OST0000 complete [ 4779.488577] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4779.494996] LustreError: Skipped 1 previous similar message [ 4779.497946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4779.510330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4779.514066] Lustre: Skipped 1 previous similar message [ 4784.098685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4789.217046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4789.220527] Lustre: Skipped 3 previous similar messages [ 4793.311300] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4793.410699] Lustre: server umount lustre-MDT0000 complete [ 4794.337150] LustreError: 310800:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4794.352155] LustreError: 310800:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4794.953982] LustreError: 310783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778351674 with bad export cookie 14064210117397776630 [ 4794.961472] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4794.963595] LustreError: 310783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4795.123109] Lustre: server umount lustre-MDT0001 complete [ 4799.466773] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 4800.977653] Key type lgssc unregistered [ 4801.160650] LNet: 315694:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4801.166077] LNetError: 315694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4801.176567] LNet: Removed LNI 192.168.204.140@tcp [ 4801.553462] Key type .llcrypt unregistered [ 4801.555975] Key type ._llcrypt unregistered [ 4835.093431] Key type ._llcrypt registered [ 4835.094541] Key type .llcrypt registered [ 4835.133666] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 4841.936187] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 4842.422873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4842.476196] alg: No test for adler32 (adler32-zlib) [ 4843.405130] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 4843.514637] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 4845.111236] Key type lgssc registered [ 4845.647893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4859.901030] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4859.912771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4861.010245] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4861.031121] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4861.065122] Lustre: lustre-MDT0000: new disk, initializing [ 4861.088525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4861.094232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4862.486181] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4867.768148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4867.825729] Lustre: 320560:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4867.866553] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4867.870147] Lustre: Skipped 1 previous similar message [ 4867.918265] Lustre: lustre-MDT0001: new disk, initializing [ 4867.955540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4867.969238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4867.974545] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4869.570910] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4873.396252] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4874.793161] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4877.130872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4877.273236] Lustre: lustre-OST0000: new disk, initializing [ 4877.275157] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4877.313239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4877.949316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4877.953068] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4878.001094] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4879.348699] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4882.803764] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4884.746508] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4884.877260] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4886.332143] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4886.396067] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4888.549744] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4888.550272] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4888.559789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4888.563468] LustreError: Skipped 1 previous similar message [ 4888.570631] Lustre: Skipped 1 previous similar message [ 4889.292981] Lustre: server umount lustre-OST0000 complete [ 4893.664729] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4893.664820] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4893.674078] Lustre: Skipped 1 previous similar message [ 4893.677036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4893.679198] Lustre: Skipped 1 previous similar message [ 4896.787377] Lustre: server umount lustre-MDT0000 complete [ 4897.876067] LustreError: 320555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778351776 with bad export cookie 8366137111411635229 [ 4897.879459] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4897.879511] LustreError: 320555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4897.986638] Lustre: server umount lustre-MDT0001 complete [ 4900.435634] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 14:36:18 (1778351778) [ 4902.530706] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 14:36:20 (1778351780) [ 4905.019726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4905.172867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4906.384138] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4909.052403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4909.178153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4910.443994] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4912.145963] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4913.249722] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4915.540053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4917.470624] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4919.148855] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4923.281413] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 4923.771675] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4924.229997] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4924.716501] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 4925.315044] Lustre: Failing over lustre-MDT0000 [ 4925.451278] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.40@tcp (stopping) [ 4925.533740] Lustre: server umount lustre-MDT0000 complete [ 4930.017356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4930.018771] LustreError: 324906:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4930.022336] Lustre: Skipped 3 previous similar messages [ 4930.028104] LustreError: 324906:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4930.029969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4930.563824] LustreError: 326363:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4935.136510] LustreError: 324909:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4935.142215] LustreError: 324909:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4938.429527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4938.482411] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4938.578787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4938.580813] Lustre: Skipped 1 previous similar message [ 4938.608781] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4940.139495] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4940.804920] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4943.850896] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4943.873878] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4943.905759] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4945.574837] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4946.272959] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4948.345057] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 4949.282429] Lustre: Failing over lustre-MDT0000 [ 4949.506662] Lustre: server umount lustre-MDT0000 complete [ 4951.058586] LustreError: 328135:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4951.069637] LustreError: 328135:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4954.082525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4954.083143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4956.164633] LustreError: 324904:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4956.177204] LustreError: 324904:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4963.670568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4963.739796] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4963.883886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4963.934947] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4965.731967] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4966.406357] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4968.937717] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4968.941722] Lustre: Skipped 2 previous similar messages [ 4968.954714] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4968.983604] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 4970.398150] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4971.069672] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4974.050850] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4974.052541] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4974.053962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4974.070049] Lustre: Skipped 4 previous similar messages [ 4979.169408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4979.174559] Lustre: Skipped 4 previous similar messages [ 4979.630537] Lustre: server umount lustre-MDT0000 complete [ 4981.400809] LustreError: 324890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778351860 with bad export cookie 8366137111411637910 [ 4981.406404] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4981.408863] LustreError: 324890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4981.577269] Lustre: server umount lustre-MDT0001 complete [ 4993.657755] Lustre: server umount lustre-OST0000 complete [ 5001.155363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5001.352312] LustreError: 330686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5001.360064] LustreError: 330686:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5001.378752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5002.936662] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5006.875376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5008.616689] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5010.969668] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5012.621217] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5015.575694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5018.387887] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5020.737338] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5026.028087] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5028.320117] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5028.323644] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5028.329106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5033.832607] Lustre: server umount lustre-OST0000 complete [ 5041.632420] Lustre: server umount lustre-MDT0000 complete [ 5041.633626] LustreError: 331844:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5041.639249] LustreError: 331844:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5043.403432] LustreError: 330667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778351922 with bad export cookie 8366137111411638631 [ 5043.405406] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5043.409492] LustreError: 330667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5043.553790] Lustre: server umount lustre-MDT0001 complete [ 5047.291196] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 5048.703801] Key type lgssc unregistered [ 5048.869426] LNet: 333863:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5048.872968] LNetError: 333863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5048.881529] LNet: Removed LNI 192.168.204.140@tcp [ 5049.332958] Key type .llcrypt unregistered [ 5049.334623] Key type ._llcrypt unregistered [ 5059.289628] Key type ._llcrypt registered [ 5059.291407] Key type .llcrypt registered [ 5059.333783] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 5059.883073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5059.908726] alg: No test for adler32 (adler32-zlib) [ 5060.774969] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 5060.865936] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 5062.455244] Key type lgssc registered [ 5063.007329] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5066.331776] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5066.342137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5067.539795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5069.241504] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5072.737281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5072.869650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5074.431896] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5076.557784] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5077.770631] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5079.963872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5080.094912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5081.901322] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5083.606985] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5092.387076] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5098.464576] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5098.470765] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5098.479185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5102.562785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5102.565552] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5102.569402] Lustre: Skipped 1 previous similar message [ 5103.734944] Lustre: server umount lustre-MDT0000 complete [ 5105.552790] LustreError: 335322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778351984 with bad export cookie 4765626243047355447 [ 5105.560108] LustreError: 335322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5105.562117] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5105.745953] Lustre: server umount lustre-MDT0001 complete [ 5117.944839] Lustre: server umount lustre-OST0000 complete [ 5122.313341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5122.464804] LustreError: 338736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5122.469703] LustreError: 338736:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5122.488807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5124.253553] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5127.650851] LustreError: 338737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5128.666673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5128.880121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5130.781831] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5133.306784] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5135.048317] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5138.351556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5138.550287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5141.221219] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5143.855443] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5148.784495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5159.398731] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5159.401422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5159.406669] Lustre: Skipped 2 previous similar messages [ 5159.410756] Lustre: Skipped 3 previous similar messages [ 5162.609113] Lustre: server umount lustre-MDT0000 complete [ 5164.410907] LustreError: 338718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778352043 with bad export cookie 4765626243047356546 [ 5164.417131] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5164.420397] LustreError: 338718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5164.512449] LustreError: 338733:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5164.513062] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5164.565238] Lustre: server umount lustre-MDT0001 complete [ 5172.399877] Lustre: server umount lustre-OST0000 complete [ 5179.249690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5179.452346] LustreError: 342308:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5179.470487] LustreError: 342308:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5179.490645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5180.882584] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5184.523101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5186.534334] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5189.127610] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5190.909983] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5194.404338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5196.948082] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5198.553248] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5204.851946] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5211.104465] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5211.109871] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5211.118993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5211.123239] Lustre: Skipped 1 previous similar message [ 5215.206613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5215.208590] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5215.210878] Lustre: Skipped 1 previous similar message [ 5216.922082] Lustre: server umount lustre-OST0000 complete [ 5220.322445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5220.327089] Lustre: Skipped 2 previous similar messages [ 5224.514625] Lustre: server umount lustre-MDT0000 complete [ 5225.440920] LustreError: 344575:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5225.456321] LustreError: 344575:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5226.453004] LustreError: 343923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778352105 with bad export cookie 4765626243047357554 [ 5226.459963] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5226.464079] LustreError: 343923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5226.655844] Lustre: server umount lustre-MDT0001 complete [ 5231.422921] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 5232.937232] Key type lgssc unregistered [ 5233.082571] LNet: 345510:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5233.086186] LNetError: 345510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5233.093684] LNet: Removed LNI 192.168.204.140@tcp [ 5233.514503] Key type .llcrypt unregistered [ 5233.516981] Key type ._llcrypt unregistered [ 5244.701147] Key type ._llcrypt registered [ 5244.702159] Key type .llcrypt registered [ 5244.742310] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 5245.306861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5245.318833] alg: No test for adler32 (adler32-zlib) [ 5246.174093] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 5246.262338] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 5247.839240] Key type lgssc registered [ 5248.367319] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5251.857992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5251.869660] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5251.875583] LustreError: 346911:0:(osd_handler.c:8532:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5251.887741] LustreError: 346911:0:(obd_config.c:843:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5251.892460] LustreError: 346911:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5251.897408] LustreError: 346911:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5251.903086] LustreError: 346911:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5255.425119] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5255.432415] LustreError: 347308:0:(osd_handler.c:8532:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5255.444725] LustreError: 347308:0:(obd_config.c:843:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5255.449664] LustreError: 347308:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5255.454562] LustreError: 347308:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5255.459906] LustreError: 347308:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5264.085310] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 5265.511468] Key type lgssc unregistered [ 5265.677799] LNet: 348292:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5265.682955] LNetError: 348292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5265.698605] LNet: Removed LNI 192.168.204.140@tcp [ 5266.176587] Key type .llcrypt unregistered [ 5266.179196] Key type ._llcrypt unregistered [ 5274.977123] Key type ._llcrypt registered [ 5274.978783] Key type .llcrypt registered [ 5275.050322] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 5282.789929] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 5283.199645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5283.206128] alg: No test for adler32 (adler32-zlib) [ 5284.125194] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 5284.247986] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 5285.839250] Key type lgssc registered [ 5286.465145] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5302.241634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5302.253840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5303.372197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5303.383049] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5303.424276] Lustre: lustre-MDT0000: new disk, initializing [ 5303.456789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5303.469150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5305.182689] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5310.669058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5310.730282] Lustre: 351387:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5310.775949] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5310.782411] Lustre: Skipped 1 previous similar message [ 5310.836352] Lustre: lustre-MDT0001: new disk, initializing [ 5310.891217] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5310.912165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5310.918464] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5312.752442] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5316.494927] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5318.139204] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5321.570956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5321.753134] Lustre: lustre-OST0000: new disk, initializing [ 5321.756887] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5321.810246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5324.561642] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5327.376956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5327.384344] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5327.454812] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5328.691859] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5330.754622] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5330.899269] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5332.894213] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5333.036427] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5337.577331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5337.577514] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5337.587382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5337.590241] LustreError: Skipped 1 previous similar message [ 5337.593967] Lustre: Skipped 1 previous similar message [ 5342.694922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5342.699028] Lustre: Skipped 1 previous similar message [ 5347.816194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5347.820354] Lustre: Skipped 1 previous similar message [ 5348.319179] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5348.393733] Lustre: server umount lustre-OST0000 complete [ 5351.904355] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5351.908488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5351.915096] Lustre: Skipped 1 previous similar message [ 5351.917780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5352.931008] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5356.192928] Lustre: server umount lustre-MDT0000 complete [ 5357.913165] LustreError: 351381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778352236 with bad export cookie 10390543241454605218 [ 5357.918892] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5357.921314] LustreError: 351381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5358.072497] Lustre: server umount lustre-MDT0001 complete [ 5361.669080] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5362.485733] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 14:44:00 (1778352240) [ 5365.622834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5365.867070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5367.413980] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5371.293158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5371.359969] LustreError: 355442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5371.368499] LustreError: 355442:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5371.485064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5373.245609] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5375.580059] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5377.154719] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5379.751451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5379.933975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5382.266964] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5384.166451] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5386.494046] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5386.561972] Lustre: lustre-OST0001: new disk, initializing [ 5386.564082] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5388.444582] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5389.638135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5389.641222] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5389.678446] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5391.858317] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5394.912501] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5394.912702] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5394.915762] LustreError: Skipped 1 previous similar message [ 5394.916290] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5394.919038] Lustre: Skipped 1 previous similar message [ 5394.920480] Lustre: Skipped 2 previous similar messages [ 5400.069563] Lustre: server umount lustre-OST0001 complete [ 5401.062713] LustreError: 357037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5401.067251] LustreError: 357037:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5406.184679] LustreError: 358388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5406.195176] LustreError: 358388:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5411.311536] LustreError: 358388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5411.322538] LustreError: 358388:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5416.423423] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5416.423747] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5416.424743] LustreError: 358388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5416.429254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5416.449815] Lustre: Skipped 4 previous similar messages [ 5418.609404] Lustre: server umount lustre-OST0000 complete [ 5426.793542] Lustre: server umount lustre-MDT0000 complete [ 5428.469766] LustreError: 355423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778352307 with bad export cookie 10390543241454606079 [ 5428.478076] LustreError: 355423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5428.478512] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5434.714986] Lustre: server umount lustre-MDT0001 complete [ 5438.107870] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 5439.360924] Key type lgssc unregistered [ 5439.520745] LNet: 359795:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5439.528345] LNetError: 359795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5439.536683] LNet: Removed LNI 192.168.204.140@tcp [ 5439.962877] Key type .llcrypt unregistered [ 5439.965048] Key type ._llcrypt unregistered [ 5462.286736] Key type ._llcrypt registered [ 5462.288680] Key type .llcrypt registered [ 5462.349793] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 5462.804742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5462.826038] alg: No test for adler32 (adler32-zlib) [ 5463.709186] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 5463.840813] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 5465.447175] Key type lgssc registered [ 5466.078698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5469.892932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5469.904455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5471.073887] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5471.086891] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5471.169323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5472.963707] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5476.605836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5476.645405] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5476.657417] Lustre: 362549:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5476.782352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5478.595843] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5481.234763] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5483.068833] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5487.072665] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5487.077807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5487.086767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5488.609661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5488.610937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5493.729509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5493.733547] Lustre: Skipped 2 previous similar messages [ 5498.335198] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5498.439917] Lustre: server umount lustre-MDT0000 complete [ 5500.086458] LustreError: 362544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778352379 with bad export cookie 7565345285971030883 [ 5500.092255] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5500.096697] LustreError: 362544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5500.265561] Lustre: server umount lustre-MDT0001 complete [ 5504.898214] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 14:46:22 (1778352382) [ 5508.180637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5508.427084] LustreError: 364834:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5508.435343] LustreError: 364834:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5508.462420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5510.451892] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5513.697612] LustreError: 364835:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5514.881876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5515.074339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5517.073466] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5519.485546] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5521.127516] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5524.344676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5524.474264] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5524.569263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5527.111941] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5529.703612] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5533.908910] Lustre: Permanently deactivating lustre-OST0000 [ 5533.914109] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5541.359162] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5542.607019] LustreError: 367579:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5542.610525] LustreError: 367579:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5542.612566] LustreError: 367579:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5543.198596] LustreError: 367627:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5543.200935] LustreError: 367627:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5549.903166] Lustre: server umount lustre-OST0000 complete [ 5550.728148] LustreError: 367876:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5550.730467] LustreError: 367876:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5553.633282] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5553.633838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5553.637131] Lustre: Skipped 1 previous similar message [ 5558.752481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5558.754416] Lustre: Skipped 2 previous similar messages [ 5561.847661] Lustre: server umount lustre-MDT0000 complete [ 5562.901248] LustreError: 364816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778352441 with bad export cookie 7565345285971031450 [ 5562.904323] LustreError: 364816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5562.906302] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5572.117790] Lustre: server umount lustre-MDT0001 complete [ 5574.722312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5575.917048] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5576.747894] LustreError: 369015:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5578.665761] Lustre: 369354:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.140@tcp [ 5579.307810] Lustre: server umount MGS complete [ 5582.761301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5582.915589] LustreError: 370077:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5582.920503] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5582.922343] Lustre: Skipped 1 previous similar message [ 5582.942793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5584.130446] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5586.754610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5586.853679] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5588.445980] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5591.049886] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5592.744828] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5593.672566] Lustre: Permanently reactivating lustre-OST0000 [ 5593.682547] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5593.687274] Lustre: Skipped 2 previous similar messages [ 5620.191231] LustreError: 371641:0:(osp_dev.c:694:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5624.305361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5624.500994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5624.505164] Lustre: Skipped 1 previous similar message [ 5627.046278] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5629.587794] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5645.280408] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5645.285636] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5645.294355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5646.313520] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5648.170902] Lustre: server umount lustre-OST0000 complete [ 5651.427298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5651.436047] Lustre: Skipped 2 previous similar messages [ 5656.183718] Lustre: server umount lustre-MDT0000 complete [ 5656.545569] LustreError: 370078:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5656.559267] LustreError: 370078:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5657.904911] LustreError: 370102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778352536 with bad export cookie 7565345285971032500 [ 5657.911410] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5657.914675] LustreError: 370102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5658.067299] Lustre: server umount lustre-MDT0001 complete [ 5661.931726] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 5663.416729] Key type lgssc unregistered [ 5663.537402] LNet: 374346:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5663.539957] LNetError: 374346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5663.546476] LNet: Removed LNI 192.168.204.140@tcp [ 5664.006207] Key type .llcrypt unregistered [ 5664.008603] Key type ._llcrypt unregistered [ 5671.954746] Key type ._llcrypt registered [ 5671.956331] Key type .llcrypt registered [ 5672.009520] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 5679.772495] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 5680.169617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5680.181432] alg: No test for adler32 (adler32-zlib) [ 5681.099791] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 5681.216700] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 5682.823133] Key type lgssc registered [ 5683.173124] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5695.614799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5695.623460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5696.746683] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5696.764761] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5696.816707] Lustre: lustre-MDT0000: new disk, initializing [ 5696.853860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5696.867134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5698.665618] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5704.228473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5704.286095] Lustre: 377441:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5704.314359] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5704.316735] Lustre: Skipped 1 previous similar message [ 5704.358215] Lustre: lustre-MDT0001: new disk, initializing [ 5704.399311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5704.421546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5704.427343] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5706.310496] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5710.363724] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5712.109209] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5715.482168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5715.659622] Lustre: lustre-OST0000: new disk, initializing [ 5715.663124] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5715.712602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5718.391956] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5720.599321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5720.607465] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5720.670874] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5722.522611] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5724.390583] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5724.528483] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5726.270104] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5726.347864] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5730.788049] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5730.789481] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5730.793037] LustreError: Skipped 1 previous similar message [ 5730.798492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5730.802149] Lustre: Skipped 1 previous similar message [ 5733.167246] Lustre: server umount lustre-OST0000 complete [ 5735.392155] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5735.395247] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5735.400124] Lustre: Skipped 1 previous similar message [ 5735.403707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5741.025752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5741.029535] Lustre: Skipped 2 previous similar messages [ 5741.179344] Lustre: server umount lustre-MDT0000 complete [ 5743.040650] LustreError: 377433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778352622 with bad export cookie 1587655968825607656 [ 5743.048412] LustreError: 377433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5743.049682] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5743.241140] Lustre: server umount lustre-MDT0001 complete [ 5747.238571] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5748.093839] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 14:50:26 (1778352626) [ 5751.432818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5751.725739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5753.411032] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5756.898124] LustreError: 381495:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5756.908806] LustreError: 381495:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5757.599435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5757.788118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5759.624581] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5761.708529] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5763.314819] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5766.311376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5766.488734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5768.940694] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5771.094409] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5771.820729] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5777.539377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5781.988879] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5781.990254] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5782.001274] Lustre: Skipped 1 previous similar message [ 5782.005229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5782.009299] Lustre: Skipped 1 previous similar message [ 5785.388666] Lustre: server umount lustre-OST0000 complete [ 5788.640410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5788.645158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5788.652382] Lustre: Skipped 1 previous similar message [ 5788.655700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5793.330183] Lustre: server umount lustre-MDT0000 complete [ 5795.060894] LustreError: 383847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778352674 with bad export cookie 1587655968825608538 [ 5795.067617] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5795.068126] LustreError: 383847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5795.223228] Lustre: server umount lustre-MDT0001 complete [ 5799.859538] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 5801.372198] Key type lgssc unregistered [ 5801.552693] LNet: 384870:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5801.557698] LNetError: 384870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5801.568580] LNet: Removed LNI 192.168.204.140@tcp [ 5802.033645] Key type .llcrypt unregistered [ 5802.035916] Key type ._llcrypt unregistered [ 5812.647885] Key type ._llcrypt registered [ 5812.650203] Key type .llcrypt registered [ 5812.717256] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 5812.893361] Key type .llcrypt unregistered [ 5812.895626] Key type ._llcrypt unregistered [ 5822.059397] Key type ._llcrypt registered [ 5822.061648] Key type .llcrypt registered [ 5822.129295] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 5822.554532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5822.601125] alg: No test for adler32 (adler32-zlib) [ 5823.495709] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 5823.629763] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 5825.239183] Key type lgssc registered [ 5825.881586] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5829.884943] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5829.899330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5831.124539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5832.944170] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5837.123314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5837.310564] LustreError: 387021:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5837.315610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5837.319249] LustreError: 387021:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5839.860427] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5841.445305] LustreError: 387025:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5841.451260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5841.453040] LustreError: 387025:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5842.158637] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5845.234979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5845.424771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5847.167558] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5855.715677] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5855.724406] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5855.724803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5855.733547] Lustre: Skipped 1 previous similar message [ 5860.835127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5860.839370] Lustre: Skipped 1 previous similar message [ 5861.488573] Lustre: server umount lustre-OST0000 complete [ 5865.954700] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5865.955150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5865.956304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5865.971758] Lustre: Skipped 2 previous similar messages [ 5869.434509] Lustre: server umount lustre-MDT0000 complete [ 5871.072840] LustreError: 387043:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5871.083523] LustreError: 387043:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5871.196714] LustreError: 387006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778352750 with bad export cookie 18107456408949345131 [ 5871.203707] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5871.204604] LustreError: 387006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5871.351203] Lustre: server umount lustre-MDT0001 complete [ 5876.036686] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 5877.437572] Key type lgssc unregistered [ 5877.627702] LNet: 390015:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5877.633001] LNetError: 390015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5877.643773] LNet: Removed LNI 192.168.204.140@tcp [ 5878.139903] Key type .llcrypt unregistered [ 5878.142260] Key type ._llcrypt unregistered [ 5890.154583] Key type ._llcrypt registered [ 5890.156975] Key type .llcrypt registered [ 5890.227199] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 5890.679909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5890.735949] alg: No test for adler32 (adler32-zlib) [ 5891.663072] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 5891.793170] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 5893.399125] Key type lgssc registered [ 5894.034991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5897.956285] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5897.967895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5913.442813] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5918.559241] LustreError: 391365:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.140@tcp: failed processing log, type 4: rc = -110 [ 5945.183244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5947.755153] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5950.345181] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5953.582291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5953.875437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5953.919513] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5955.624795] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5959.138592] LustreError: 391391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5959.149760] LustreError: 391391:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5959.716244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5959.921281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5961.700117] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5964.173422] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5965.875678] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5969.376417] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5969.381636] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5969.390608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5970.400907] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5970.406133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5970.415397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5971.428507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5971.432634] Lustre: Skipped 1 previous similar message [ 5974.756185] Lustre: server umount lustre-OST0000 complete [ 5976.544893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5976.553179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5976.556130] Lustre: Skipped 1 previous similar message [ 5981.664897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5981.669032] Lustre: Skipped 1 previous similar message [ 5982.703737] Lustre: server umount lustre-MDT0000 complete [ 5984.473891] LustreError: 391371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778352863 with bad export cookie 18079800992219726923 [ 5984.480124] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5984.482147] LustreError: 391371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5984.627352] Lustre: server umount lustre-MDT0001 complete [ 5989.238667] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 5990.663705] Key type lgssc unregistered [ 5990.822639] LNet: 394579:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5990.827817] LNetError: 394579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5990.838517] LNet: Removed LNI 192.168.204.140@tcp [ 5991.283930] Key type .llcrypt unregistered [ 5991.286346] Key type ._llcrypt unregistered [ 6002.864762] Key type ._llcrypt registered [ 6002.867040] Key type .llcrypt registered [ 6002.930206] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 6003.244183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6003.251163] alg: No test for adler32 (adler32-zlib) [ 6004.125513] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 6004.226708] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 6005.831213] Key type lgssc registered [ 6006.433931] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6010.235355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6010.247331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6011.490870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6013.163653] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6017.224490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6017.383712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6019.020993] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6021.582966] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6023.280512] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6026.486633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6026.643545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6029.143348] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6031.752285] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6036.913884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6042.083245] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6042.088412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6042.091533] Lustre: Skipped 2 previous similar messages [ 6042.095650] Lustre: Skipped 1 previous similar message [ 6044.221608] Lustre: server umount lustre-MDT0000 complete [ 6045.322932] LustreError: 395935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.40@tcp arrived at 1778352924 with bad export cookie 14917976282870677946 [ 6046.278450] LustreError: 395936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778352925 with bad export cookie 14917976282870677295 [ 6046.286304] LustreError: 395936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6046.289806] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6047.200957] LustreError: 396665:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6047.202183] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6047.210816] LustreError: 396665:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6047.219888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6047.227663] Lustre: Skipped 1 previous similar message [ 6051.232856] LustreError: 395955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6051.233638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6051.243583] LustreError: 395955:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6052.466982] Lustre: server umount lustre-OST0000 complete [ 6055.800109] Lustre: server umount lustre-MDT0001 complete [ 6060.598538] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 6062.085406] Key type lgssc unregistered [ 6062.265756] LNet: 399284:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6062.271047] LNetError: 399284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6062.281753] LNet: Removed LNI 192.168.204.140@tcp [ 6062.772935] Key type .llcrypt unregistered [ 6062.775861] Key type ._llcrypt unregistered [ 6075.319139] Key type ._llcrypt registered [ 6075.321444] Key type .llcrypt registered [ 6075.392356] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 6075.898847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6075.911101] alg: No test for adler32 (adler32-zlib) [ 6076.814454] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 6076.949862] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 6078.559185] Key type lgssc registered [ 6079.192275] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6083.153991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6083.165984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6084.425173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6086.366992] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6090.586725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6090.803164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6092.649224] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6095.384780] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6097.288840] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6100.632418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6100.818659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6103.378782] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6106.011515] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6109.040379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6113.219828] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6113.228633] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6113.232973] Lustre: Skipped 1 previous similar message [ 6117.382466] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.40@tcp (stopping) [ 6119.415900] Lustre: server umount lustre-MDT0001 complete [ 6123.490593] LustreError: 401383:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6123.491295] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6123.500429] LustreError: 401383:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6123.508197] Lustre: Skipped 1 previous similar message [ 6123.508909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6123.519463] Lustre: Skipped 2 previous similar messages [ 6127.915902] Lustre: server umount lustre-OST0000 complete [ 6128.611574] LustreError: 400661:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6129.926301] Lustre: server umount lustre-MDT0000 complete [ 6136.184285] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 6137.842608] Key type lgssc unregistered [ 6138.027748] LNet: 403990:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6138.032510] LNetError: 403990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6138.042749] LNet: Removed LNI 192.168.204.140@tcp [ 6138.542377] Key type .llcrypt unregistered [ 6138.544793] Key type ._llcrypt unregistered [ 6150.037564] Key type ._llcrypt registered [ 6150.039520] Key type .llcrypt registered [ 6150.115937] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 6157.921228] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 6158.247830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6158.280140] alg: No test for adler32 (adler32-zlib) [ 6159.179419] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 6159.309808] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 6160.919164] Key type lgssc registered [ 6161.551534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6180.246434] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 6187.495767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6187.506794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6188.653891] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6188.672220] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6188.719156] Lustre: lustre-MDT0000: new disk, initializing [ 6188.754465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6188.763642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6190.731299] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6198.832131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6199.007065] Lustre: lustre-OST0000: new disk, initializing [ 6199.010938] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6199.014915] Lustre: Skipped 1 previous similar message [ 6199.059125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6201.776208] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6203.032490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6203.038804] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6203.053340] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6205.853610] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6213.559797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6213.642955] Lustre: 407926:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6213.700405] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6213.754661] Lustre: lustre-MDT0001: new disk, initializing [ 6213.828238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6213.854606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6213.861629] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6215.762172] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6220.507349] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6220.640382] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6222.649461] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6222.785596] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6225.379405] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6225.382789] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6225.385174] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6225.392567] Lustre: Skipped 1 previous similar message [ 6225.399430] Lustre: Skipped 1 previous similar message [ 6226.454456] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.40@tcp (stopping) [ 6227.253368] Lustre: server umount lustre-MDT0001 complete [ 6235.623488] LustreError: 407933:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6235.633337] LustreError: 407933:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6240.738407] LustreError: 407939:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6240.738831] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6240.757435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6245.858154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6245.860179] LustreError: 407938:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6246.578457] Lustre: server umount lustre-OST0000 complete [ 6254.780370] Lustre: server umount lustre-MDT0000 complete [ 6260.887674] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 6262.484734] Key type lgssc unregistered [ 6262.664648] LNet: 412494:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6262.669997] LNetError: 412494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6262.678687] LNet: Removed LNI 192.168.204.140@tcp [ 6263.152945] Key type .llcrypt unregistered [ 6263.155281] Key type ._llcrypt unregistered [ 6281.425395] Key type ._llcrypt registered [ 6281.427218] Key type .llcrypt registered [ 6281.494046] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 6281.668523] Key type .llcrypt unregistered [ 6281.671051] Key type ._llcrypt unregistered [ 6290.274719] Key type ._llcrypt registered [ 6290.276590] Key type .llcrypt registered [ 6290.342974] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 6290.777074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6290.832554] alg: No test for adler32 (adler32-zlib) [ 6291.707193] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 6291.832848] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 6293.431166] Key type lgssc registered [ 6294.046482] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6307.380632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6307.392428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6308.510066] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6308.526363] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6308.572432] Lustre: lustre-MDT0000: new disk, initializing [ 6308.608430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6308.619165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6310.630624] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6316.352665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6316.427052] Lustre: 415811:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6316.505149] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6316.509210] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6316.512956] Lustre: Skipped 1 previous similar message [ 6316.565563] Lustre: lustre-MDT0001: new disk, initializing [ 6316.620535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6316.643769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6316.651178] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6318.556426] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6322.716903] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6324.543570] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6327.923572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6328.095810] Lustre: lustre-OST0000: new disk, initializing [ 6328.099912] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6328.153003] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6330.815412] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6331.406761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6331.412412] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6331.462805] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6334.938730] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6341.606265] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6341.608775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6341.614342] Lustre: Skipped 2 previous similar messages [ 6341.618023] Lustre: Skipped 1 previous similar message [ 6344.012467] Lustre: server umount lustre-MDT0000 complete [ 6345.861811] LustreError: 415802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778353224 with bad export cookie 12425304225826254655 [ 6345.868129] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6345.869173] LustreError: 415802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6346.015535] Lustre: server umount lustre-MDT0001 complete [ 6357.119210] Lustre: server umount lustre-OST0000 complete [ 6362.338220] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 15:00:40 (1778353240) [ 6365.826121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6366.045719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6367.721187] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6371.297594] LustreError: 419465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6371.309184] LustreError: 419465:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6371.744993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6371.908692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6373.641774] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6376.140398] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6377.724336] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6393.892629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6394.008059] Lustre: Found index 0 for lustre-OST0000, updating log [ 6394.108435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6396.645600] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6399.186494] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6411.500066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6416.866208] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6416.866441] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6416.875643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6416.879750] LustreError: Skipped 1 previous similar message [ 6416.886703] Lustre: Skipped 2 previous similar messages [ 6418.661446] Lustre: server umount lustre-OST0000 complete [ 6431.873691] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6431.875373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6431.882237] Lustre: Skipped 2 previous similar messages [ 6431.886466] Lustre: Skipped 2 previous similar messages [ 6437.344674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6437.349344] Lustre: Skipped 1 previous similar message [ 6438.071072] Lustre: server umount lustre-MDT0000 complete [ 6439.849584] LustreError: 419446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778353318 with bad export cookie 12425304225826256048 [ 6439.853926] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6439.857658] LustreError: 419446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6440.003508] Lustre: server umount lustre-MDT0001 complete [ 6443.835276] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 15:02:01 (1778353321) [ 6452.639368] LNet: 1 local NIs in recovery (showing 1): 192.168.204.140@tcp [ 7031.097268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7031.312972] LustreError: 425564:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7031.348829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7032.963563] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7036.385894] LustreError: 425565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7036.948420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7037.148445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7038.759832] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7041.249124] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7042.931828] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7059.108261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7059.241901] Lustre: Found index 0 for lustre-OST0000, updating log [ 7059.392664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7061.944822] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7064.399524] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7069.667103] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7069.672391] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7069.681620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7071.458448] Lustre: server umount lustre-OST0000 complete [ 7073.248319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7073.253678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7073.262418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7074.275672] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7075.297131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7075.301175] Lustre: Skipped 3 previous similar messages [ 7079.346605] Lustre: server umount lustre-MDT0000 complete [ 7079.393761] LustreError: 425564:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7079.404898] LustreError: 425564:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7081.105820] LustreError: 425546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778353960 with bad export cookie 12425304225826257119 [ 7081.112198] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7081.113672] LustreError: 425546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7081.264780] Lustre: server umount lustre-MDT0001 complete [ 7092.639420] LNet: 1 local NIs in recovery (showing 1): 192.168.204.140@tcp [ 7101.171570] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 7105.826580] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 7123.540256] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 15:13:21 (1778354001) [ 7131.580127] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 7133.178988] Key type lgssc unregistered [ 7133.361965] LNet: 431833:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7133.367644] LNetError: 431833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7189.733603] LNet: Removed LNI 192.168.204.140@tcp [ 7190.178877] Key type .llcrypt unregistered [ 7190.181242] Key type ._llcrypt unregistered [ 7198.487980] Key type ._llcrypt registered [ 7198.490346] Key type .llcrypt registered [ 7198.564781] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 7199.078896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7199.091731] alg: No test for adler32 (adler32-zlib) [ 7199.988151] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 7200.121241] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 7201.735152] Key type lgssc registered [ 7202.384440] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7204.660707] LNet: Added LNI 192.168.204.140@tcp100 [8/256/0/180] [ 7205.097198] LNet: Added LNI 192.168.204.140@tcp101 [8/256/0/180] [ 7205.529297] LNet: Added LNI 192.168.204.140@tcp102 [8/256/0/180] [ 7205.941135] LNet: Added LNI 192.168.204.140@tcp103 [8/256/0/180] [ 7206.324358] LNet: Added LNI 192.168.204.140@tcp104 [8/256/0/180] [ 7206.762133] LNet: Added LNI 192.168.204.140@tcp105 [8/256/0/180] [ 7207.195395] LNet: Added LNI 192.168.204.140@tcp106 [8/256/0/180] [ 7207.633773] LNet: Added LNI 192.168.204.140@tcp107 [8/256/0/180] [ 7208.059452] LNet: Added LNI 192.168.204.140@tcp108 [8/256/0/180] [ 7208.494330] LNet: Added LNI 192.168.204.140@tcp109 [8/256/0/180] [ 7208.935746] LNet: Added LNI 192.168.204.140@tcp110 [8/256/0/180] [ 7209.352229] LNet: Added LNI 192.168.204.140@tcp111 [8/256/0/180] [ 7209.794998] LNet: Added LNI 192.168.204.140@tcp112 [8/256/0/180] [ 7210.226908] LNet: Added LNI 192.168.204.140@tcp113 [8/256/0/180] [ 7210.657723] LNet: Added LNI 192.168.204.140@tcp114 [8/256/0/180] [ 7211.094851] LNet: Added LNI 192.168.204.140@tcp115 [8/256/0/180] [ 7211.530698] LNet: Added LNI 192.168.204.140@tcp116 [8/256/0/180] [ 7211.968270] LNet: Added LNI 192.168.204.140@tcp117 [8/256/0/180] [ 7212.402699] LNet: Added LNI 192.168.204.140@tcp118 [8/256/0/180] [ 7212.836723] LNet: Added LNI 192.168.204.140@tcp119 [8/256/0/180] [ 7213.269983] LNet: Added LNI 192.168.204.140@tcp120 [8/256/0/180] [ 7213.695828] LNet: Added LNI 192.168.204.140@tcp121 [8/256/0/180] [ 7214.132986] LNet: Added LNI 192.168.204.140@tcp122 [8/256/0/180] [ 7214.573685] LNet: Added LNI 192.168.204.140@tcp123 [8/256/0/180] [ 7215.006651] LNet: Added LNI 192.168.204.140@tcp124 [8/256/0/180] [ 7215.444186] LNet: Added LNI 192.168.204.140@tcp125 [8/256/0/180] [ 7215.880721] LNet: Added LNI 192.168.204.140@tcp126 [8/256/0/180] [ 7216.311124] LNet: Added LNI 192.168.204.140@tcp127 [8/256/0/180] [ 7216.743323] LNet: Added LNI 192.168.204.140@tcp128 [8/256/0/180] [ 7217.175562] LNet: Added LNI 192.168.204.140@tcp129 [8/256/0/180] [ 7217.616556] LNet: Added LNI 192.168.204.140@tcp130 [8/256/0/180] [ 7218.050078] LNet: Added LNI 192.168.204.140@tcp131 [8/256/0/180] [ 7218.462565] LNet: Added LNI 192.168.204.140@tcp132 [8/256/0/180] [ 7218.901110] LNet: Added LNI 192.168.204.140@tcp133 [8/256/0/180] [ 7219.334656] LNet: Added LNI 192.168.204.140@tcp134 [8/256/0/180] [ 7219.765258] LNet: Added LNI 192.168.204.140@tcp135 [8/256/0/180] [ 7222.969560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7222.980839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7224.049761] Lustre: 434812:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.140@tcp: can't add NID 192.168.204.140@tcp130: rc = -75 [ 7224.112305] LNetError: 434836:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.140 [ 7224.117219] LNetError: Protocol error connecting to 192.168.204.140@tcp118 at host 192.168.204.140:988: is it running a compatible version of Lustre? [ 7224.120282] LNetError: Skipped 1 previous similar message [ 7224.169976] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7224.185938] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7224.227640] Lustre: lustre-MDT0000: new disk, initializing [ 7224.271571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7224.282100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7224.735173] LNet: 1 local NIs in recovery (showing 1): 192.168.204.140@tcp122 [ 7225.128569] LNetError: 432274:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.140 [ 7225.135038] LNetError: 432274:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Skipped 1 previous similar message [ 7225.140191] LNetError: Protocol error connecting to 192.168.204.140@tcp100 at host 192.168.204.140:988: is it running a compatible version of Lustre? [ 7226.181815] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7227.808703] LNetError: 434834:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.140 [ 7227.815275] LNetError: 434834:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Skipped 1 previous similar message [ 7227.820581] LNetError: Protocol error connecting to 192.168.204.140@tcp122 at host 192.168.204.140:988: is it running a compatible version of Lustre? [ 7227.827567] LNetError: Skipped 1 previous similar message [ 7231.778092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7231.861353] Lustre: 434829:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7231.905656] LNetError: 434839:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.140 [ 7231.910417] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7231.911764] LNetError: Protocol error connecting to 192.168.204.140@tcp122 at host 192.168.204.140:988: is it running a compatible version of Lustre? [ 7231.915420] Lustre: Skipped 1 previous similar message [ 7231.958208] Lustre: lustre-MDT0001: new disk, initializing [ 7231.977158] Lustre: 435615:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.140@tcp: can't add NID 192.168.204.140@tcp130: rc = -75 [ 7231.983432] Lustre: 435615:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7232.059390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7232.081199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7232.088481] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7233.988349] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7238.028208] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7238.662397] Lustre: 436146:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.140@tcp: can't add NID 192.168.204.140@tcp130: rc = -75 [ 7238.668756] Lustre: 436146:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7239.072678] LNetError: 434841:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.140 [ 7239.079184] LNetError: Protocol error connecting to 192.168.204.140@tcp122 at host 192.168.204.140:988: is it running a compatible version of Lustre? [ 7239.782279] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7243.137465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7243.306116] Lustre: lustre-OST0000: new disk, initializing [ 7243.309240] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7243.362938] Lustre: 436583:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.140@tcp: can't add NID 192.168.204.140@tcp130: rc = -75 [ 7243.368186] Lustre: 436583:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 7243.392811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7244.323514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7244.329653] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7244.398246] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7246.078521] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7250.163901] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7254.497485] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7254.502683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7254.511640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7255.016146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7255.016291] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7255.024958] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7255.456713] LNetError: 432275:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.140 [ 7255.463422] LNetError: Protocol error connecting to 192.168.204.140@tcp122 at host 192.168.204.140:988: is it running a compatible version of Lustre? [ 7259.758654] Lustre: server umount lustre-OST0000 complete [ 7262.688476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7262.693712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7262.702652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7262.706553] Lustre: Skipped 1 previous similar message [ 7265.249361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7265.250928] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7267.778796] Lustre: server umount lustre-MDT0000 complete [ 7269.599394] LustreError: 434871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778354148 with bad export cookie 10068503708584581887 [ 7269.607057] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7269.607175] LustreError: 434871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7269.786416] Lustre: server umount lustre-MDT0001 complete [ 7273.888698] LNetError: 434836:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.140 [ 7273.895627] LNetError: 434836:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Skipped 2 previous similar messages [ 7273.900834] LNetError: Protocol error connecting to 192.168.204.140@tcp118 at host 192.168.204.140:988: is it running a compatible version of Lustre? [ 7273.907849] LNetError: Skipped 2 previous similar messages [ 7274.926430] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 7276.533342] Key type lgssc unregistered [ 7276.746444] LNet: 438359:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7276.752851] LNetError: 438359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7276.757979] LNet: Removed LNI 192.168.204.140@tcp [ 7277.791197] LNet: Removed LNI 192.168.204.140@tcp100 [ 7277.794031] LNetError: 438359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7277.798827] LNetError: 438359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7277.803925] LNet: Removed LNI 192.168.204.140@tcp101 [ 7278.815183] LNet: Removed LNI 192.168.204.140@tcp102 [ 7278.817975] LNetError: 438359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7278.822891] LNetError: 438359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7278.828166] LNet: Removed LNI 192.168.204.140@tcp103 [ 7279.839144] LNet: Removed LNI 192.168.204.140@tcp104 [ 7280.863160] LNet: Removed LNI 192.168.204.140@tcp105 [ 7280.865931] LNetError: 438359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7280.870826] LNetError: 438359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7280.876191] LNet: Removed LNI 192.168.204.140@tcp106 [ 7281.887207] LNet: Removed LNI 192.168.204.140@tcp107 [ 7282.911209] LNet: Removed LNI 192.168.204.140@tcp108 [ 7283.935229] LNet: Removed LNI 192.168.204.140@tcp109 [ 7284.959225] LNet: Removed LNI 192.168.204.140@tcp110 [ 7284.961922] LNetError: 438359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7284.966840] LNetError: 438359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7284.972292] LNet: Removed LNI 192.168.204.140@tcp111 [ 7285.983164] LNet: Removed LNI 192.168.204.140@tcp112 [ 7287.007234] LNet: Removed LNI 192.168.204.140@tcp113 [ 7288.031172] LNet: Removed LNI 192.168.204.140@tcp114 [ 7289.055182] LNet: Removed LNI 192.168.204.140@tcp115 [ 7290.079260] LNet: Removed LNI 192.168.204.140@tcp116 [ 7291.103242] LNet: Removed LNI 192.168.204.140@tcp117 [ 7292.127276] LNet: Removed LNI 192.168.204.140@tcp118 [ 7293.151206] LNet: Removed LNI 192.168.204.140@tcp119 [ 7293.153874] LNetError: 438359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7293.158898] LNetError: 438359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 7293.164468] LNet: Removed LNI 192.168.204.140@tcp120 [ 7294.175228] LNet: Removed LNI 192.168.204.140@tcp121 [ 7294.177950] LNet: Removed LNI 192.168.204.140@tcp122 [ 7294.180663] LNet: Removed LNI 192.168.204.140@tcp123 [ 7294.183350] LNet: Removed LNI 192.168.204.140@tcp124 [ 7294.185990] LNet: Removed LNI 192.168.204.140@tcp125 [ 7294.188554] LNet: Removed LNI 192.168.204.140@tcp126 [ 7294.191061] LNet: Removed LNI 192.168.204.140@tcp127 [ 7294.193596] LNet: Removed LNI 192.168.204.140@tcp128 [ 7294.196172] LNet: Removed LNI 192.168.204.140@tcp129 [ 7294.198645] LNet: Removed LNI 192.168.204.140@tcp130 [ 7294.201198] LNet: Removed LNI 192.168.204.140@tcp131 [ 7294.203556] LNet: Removed LNI 192.168.204.140@tcp132 [ 7294.206107] LNet: Removed LNI 192.168.204.140@tcp133 [ 7294.208663] LNet: Removed LNI 192.168.204.140@tcp134 [ 7294.218215] LNet: Removed LNI 192.168.204.140@tcp135 [ 7294.651268] Key type .llcrypt unregistered [ 7294.653540] Key type ._llcrypt unregistered [ 7305.441552] Key type ._llcrypt registered [ 7305.443328] Key type .llcrypt registered [ 7305.510982] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 7313.633123] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 7314.174098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7314.180691] alg: No test for adler32 (adler32-zlib) [ 7315.073232] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 7315.190943] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 7316.783162] Key type lgssc registered [ 7317.404633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7334.642290] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 15:16:52 (1778354212) [ 7343.180351] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 7344.787973] Key type lgssc unregistered [ 7344.969777] LNet: 442371:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7344.974632] LNetError: 442371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7344.984771] LNet: Removed LNI 192.168.204.140@tcp [ 7345.462627] Key type .llcrypt unregistered [ 7345.464834] Key type ._llcrypt unregistered [ 7354.263101] Key type ._llcrypt registered [ 7354.265042] Key type .llcrypt registered [ 7354.338919] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 7354.851978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7354.869752] alg: No test for adler32 (adler32-zlib) [ 7355.797030] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 7355.925225] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 7357.535162] Key type lgssc registered [ 7358.166825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7363.043475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7363.054516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7364.197112] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7364.216595] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7364.276283] Lustre: lustre-MDT0000: new disk, initializing [ 7364.312225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7364.322180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7366.192687] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7371.884856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7371.958178] Lustre: 443543:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7372.004864] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7372.008886] Lustre: Skipped 1 previous similar message [ 7372.072963] Lustre: lustre-MDT0001: new disk, initializing [ 7372.136425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7372.158249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7372.165703] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7374.003665] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7378.139210] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7379.928447] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7383.265744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7383.437068] Lustre: lustre-OST0000: new disk, initializing [ 7383.440985] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7383.493506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7386.260297] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7390.294213] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7390.736826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7390.743970] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7390.803239] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7392.083810] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing lnet_if_list [ 7394.382897] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7394.403466] LNet: Added LNI 192.168.204.140@tcp10 [8/256/0/180] [ 7397.100330] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7397.120171] LNet: Added LNI 192.168.204.140@tcp11 [8/256/0/180] [ 7399.145842] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7399.166447] LNet: Added LNI 192.168.204.140@tcp12 [8/256/0/180] [ 7401.146615] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7401.163992] LNet: Added LNI 192.168.204.140@tcp13 [8/256/0/180] [ 7403.080679] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7403.096100] LNet: Added LNI 192.168.204.140@tcp14 [8/256/0/180] [ 7405.173324] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7405.193683] LNet: Added LNI 192.168.204.140@tcp15 [8/256/0/180] [ 7407.339271] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7407.359622] LNet: Added LNI 192.168.204.140@tcp16 [8/256/0/180] [ 7409.575172] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7409.595265] LNet: Added LNI 192.168.204.140@tcp17 [8/256/0/180] [ 7411.831364] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7411.851044] LNet: Added LNI 192.168.204.140@tcp18 [8/256/0/180] [ 7414.203672] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7414.223987] LNet: Added LNI 192.168.204.140@tcp19 [8/256/0/180] [ 7416.619659] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7416.633197] LNet: Added LNI 192.168.204.140@tcp20 [8/256/0/180] [ 7419.116981] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7419.136933] LNet: Added LNI 192.168.204.140@tcp21 [8/256/0/180] [ 7421.581760] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7421.599533] LNet: Added LNI 192.168.204.140@tcp22 [8/256/0/180] [ 7424.160807] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7424.180271] LNet: Added LNI 192.168.204.140@tcp23 [8/256/0/180] [ 7426.865966] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7426.885974] LNet: Added LNI 192.168.204.140@tcp24 [8/256/0/180] [ 7429.545247] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7429.564783] LNet: Added LNI 192.168.204.140@tcp25 [8/256/0/180] [ 7432.167569] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7432.186154] LNet: Added LNI 192.168.204.140@tcp26 [8/256/0/180] [ 7434.925251] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7434.945464] LNet: Added LNI 192.168.204.140@tcp27 [8/256/0/180] [ 7437.593234] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7437.613071] LNet: Added LNI 192.168.204.140@tcp28 [8/256/0/180] [ 7440.593436] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7440.613898] LNet: Added LNI 192.168.204.140@tcp29 [8/256/0/180] [ 7443.606044] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7443.624765] LNet: Added LNI 192.168.204.140@tcp30 [8/256/0/180] [ 7446.683925] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7446.703923] LNet: Added LNI 192.168.204.140@tcp31 [8/256/0/180] [ 7449.472622] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7449.493685] LNet: Added LNI 192.168.204.140@tcp32 [8/256/0/180] [ 7452.669651] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7452.686294] LNet: Added LNI 192.168.204.140@tcp33 [8/256/0/180] [ 7456.053648] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7456.078156] LNet: Added LNI 192.168.204.140@tcp34 [8/256/0/180] [ 7459.394635] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7459.417761] LNet: Added LNI 192.168.204.140@tcp35 [8/256/0/180] [ 7463.017306] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7463.038036] LNet: Added LNI 192.168.204.140@tcp36 [8/256/0/180] [ 7466.517488] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7466.536429] LNet: Added LNI 192.168.204.140@tcp37 [8/256/0/180] [ 7470.036233] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7470.056668] LNet: Added LNI 192.168.204.140@tcp38 [8/256/0/180] [ 7473.664388] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7473.682488] LNet: Added LNI 192.168.204.140@tcp39 [8/256/0/180] [ 7477.415422] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7477.436387] LNet: Added LNI 192.168.204.140@tcp40 [8/256/0/180] [ 7481.237754] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7481.258395] LNet: Added LNI 192.168.204.140@tcp41 [8/256/0/180] [ 7485.146633] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7485.165839] LNet: Added LNI 192.168.204.140@tcp42 [8/256/0/180] [ 7489.118244] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7489.134605] LNet: Added LNI 192.168.204.140@tcp43 [8/256/0/180] [ 7493.069635] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7493.091898] LNet: Added LNI 192.168.204.140@tcp44 [8/256/0/180] [ 7497.250390] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7497.270814] LNet: Added LNI 192.168.204.140@tcp45 [8/256/0/180] [ 7501.315361] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7501.331979] LNet: Added LNI 192.168.204.140@tcp46 [8/256/0/180] [ 7505.314175] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7505.329505] LNet: Added LNI 192.168.204.140@tcp47 [8/256/0/180] [ 7509.288914] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7509.306514] LNet: Added LNI 192.168.204.140@tcp48 [8/256/0/180] [ 7513.512413] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7513.532119] LNet: Added LNI 192.168.204.140@tcp49 [8/256/0/180] [ 7517.747806] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7517.765612] LNet: Added LNI 192.168.204.140@tcp50 [8/256/0/180] [ 7528.933206] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7528.936599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7528.941692] Lustre: Skipped 2 previous similar messages [ 7528.945710] Lustre: Skipped 1 previous similar message [ 7533.520878] Lustre: server umount lustre-MDT0000 complete [ 7535.338125] LustreError: 443535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778354414 with bad export cookie 11123894402235334050 [ 7535.345988] LustreError: 443535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7535.346178] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7535.514403] Lustre: server umount lustre-MDT0001 complete [ 7545.146957] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 7549.296695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7555.551204] Lustre: 442818:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778354418/real 1778354418] req@00000000aa845a65 x1864739569373056/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778354434 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7555.567474] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7556.575301] Lustre: 442819:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778354419/real 1778354419] req@000000005822feb9 x1864739569373184/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778354435 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7560.671198] Lustre: 442816:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778354423/real 1778354423] req@00000000e39a8da2 x1864739569373312/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778354439 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7560.735593] LustreError: 442815:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000cb64cf37 x1864739569374208/t0(0) o250->MGC192.168.204.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7560.914534] LustreError: 445277:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7560.926630] LustreError: 445277:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7560.947631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7562.834163] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7564.964485] LustreError: 445278:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7565.983232] Lustre: 442818:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778354429/real 1778354429] req@00000000ca9425d4 x1864739569373696/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778354445 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7565.985817] LustreError: 468698:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7565.999748] Lustre: 442818:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7566.015624] LustreError: 468698:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7567.119637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7567.348236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7569.206292] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7571.423275] Lustre: 442819:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778354434/real 1778354434] req@000000003910e4d7 x1864739569373824/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778354450 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7571.765749] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7572.463975] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7573.473343] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7584.882478] Lustre: server umount lustre-OST0000 complete [ 7587.812172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7587.812573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7587.813929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7587.829182] Lustre: Skipped 1 previous similar message [ 7592.929127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7592.933632] Lustre: Skipped 2 previous similar messages [ 7598.049284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7598.053811] Lustre: Skipped 1 previous similar message [ 7601.119206] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7601.210532] Lustre: server umount lustre-MDT0000 complete [ 7602.975234] Lustre: 442819:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778354466/real 1778354466] req@000000002019c928 x1864739569393536/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1778354482 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7602.991896] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7603.002810] LustreError: 468703:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7603.017632] LustreError: 468703:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7603.032923] LustreError: 445283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778354482 with bad export cookie 11123894402235336206 [ 7603.038191] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7603.043424] LustreError: 445283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7603.195748] Lustre: server umount lustre-MDT0001 complete [ 7607.989834] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 7609.594861] Key type lgssc unregistered [ 7609.813666] LNet: 471234:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7609.819845] LNetError: 471234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7609.825236] LNet: Removed LNI 192.168.204.140@tcp [ 7609.828332] LNet: Removed LNI 192.168.204.140@tcp10 [ 7609.831126] LNet: Removed LNI 192.168.204.140@tcp11 [ 7609.833786] LNet: Removed LNI 192.168.204.140@tcp12 [ 7609.836463] LNet: Removed LNI 192.168.204.140@tcp13 [ 7609.839263] LNet: Removed LNI 192.168.204.140@tcp14 [ 7609.841726] LNet: Removed LNI 192.168.204.140@tcp15 [ 7609.844368] LNet: Removed LNI 192.168.204.140@tcp16 [ 7609.846986] LNet: Removed LNI 192.168.204.140@tcp17 [ 7609.849454] LNet: Removed LNI 192.168.204.140@tcp18 [ 7609.851862] LNet: Removed LNI 192.168.204.140@tcp19 [ 7609.855257] LNet: Removed LNI 192.168.204.140@tcp20 [ 7609.857570] LNet: Removed LNI 192.168.204.140@tcp21 [ 7609.860100] LNet: Removed LNI 192.168.204.140@tcp22 [ 7609.862388] LNet: Removed LNI 192.168.204.140@tcp23 [ 7609.864649] LNet: Removed LNI 192.168.204.140@tcp24 [ 7609.866908] LNet: Removed LNI 192.168.204.140@tcp25 [ 7609.869232] LNet: Removed LNI 192.168.204.140@tcp26 [ 7609.871541] LNet: Removed LNI 192.168.204.140@tcp27 [ 7609.873719] LNet: Removed LNI 192.168.204.140@tcp28 [ 7609.875865] LNet: Removed LNI 192.168.204.140@tcp29 [ 7609.877944] LNet: Removed LNI 192.168.204.140@tcp30 [ 7609.880099] LNet: Removed LNI 192.168.204.140@tcp31 [ 7609.882103] LNet: Removed LNI 192.168.204.140@tcp32 [ 7609.884091] LNet: Removed LNI 192.168.204.140@tcp33 [ 7609.886110] LNet: Removed LNI 192.168.204.140@tcp34 [ 7609.888093] LNet: Removed LNI 192.168.204.140@tcp35 [ 7609.890104] LNet: Removed LNI 192.168.204.140@tcp36 [ 7609.891823] LNet: Removed LNI 192.168.204.140@tcp37 [ 7609.893567] LNet: Removed LNI 192.168.204.140@tcp38 [ 7609.895432] LNet: Removed LNI 192.168.204.140@tcp39 [ 7609.897330] LNet: Removed LNI 192.168.204.140@tcp40 [ 7609.899104] LNet: Removed LNI 192.168.204.140@tcp41 [ 7609.900601] LNet: Removed LNI 192.168.204.140@tcp42 [ 7609.902249] LNet: Removed LNI 192.168.204.140@tcp43 [ 7609.903750] LNet: Removed LNI 192.168.204.140@tcp44 [ 7609.905377] LNet: Removed LNI 192.168.204.140@tcp45 [ 7609.907040] LNet: Removed LNI 192.168.204.140@tcp46 [ 7609.908550] LNet: Removed LNI 192.168.204.140@tcp47 [ 7609.910110] LNet: Removed LNI 192.168.204.140@tcp48 [ 7609.911616] LNet: Removed LNI 192.168.204.140@tcp49 [ 7609.917475] LNet: Removed LNI 192.168.204.140@tcp50 [ 7610.332459] Key type .llcrypt unregistered [ 7610.334661] Key type ._llcrypt unregistered [ 7620.433666] Key type ._llcrypt registered [ 7620.435844] Key type .llcrypt registered [ 7620.502255] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 7620.920383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7620.931530] alg: No test for adler32 (adler32-zlib) [ 7621.816581] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 7621.923881] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 7623.535154] Key type lgssc registered [ 7624.167117] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7628.095818] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7628.107402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7629.374128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7631.338818] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7635.596520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7635.812172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7637.737312] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7640.416909] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7642.240577] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7645.608200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7645.809889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7648.593870] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7651.290321] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7654.002151] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7666.577964] loop0: detected capacity change from 0 to 204800000 [ 7666.674238] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7666.678702] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7666.683740] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7669.623917] loop0: detected capacity change from 0 to 204800000 [ 7669.766140] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7669.773481] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7669.780973] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7672.352123] loop0: detected capacity change from 0 to 204800000 [ 7675.028955] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7675.100572] Lustre: fs15246-OST0000: new disk, initializing [ 7675.104675] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7675.136709] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7677.715542] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7682.926197] loop1: detected capacity change from 0 to 204800000 [ 7685.590335] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7685.649395] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7685.689144] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7685.741904] Lustre: fs15246-MDT0000: new disk, initializing [ 7685.826972] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7685.840732] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7685.914503] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7686.944127] Lustre: 476844:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000895f0991 x1864739848225920/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1778354581 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7687.730819] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7695.010721] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7695.017104] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7695.028158] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7702.511204] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7702.523508] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7703.984411] Lustre: server umount fs15246-MDT0000 complete [ 7707.627924] LustreError: 472454:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7707.637718] LustreError: 472454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7708.442277] Lustre: server umount fs15246-OST0000 complete [ 7715.808398] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7715.813187] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7715.820787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7716.833174] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7716.838418] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7716.847272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7719.350737] Lustre: server umount lustre-OST0000 complete [ 7722.979258] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7722.979850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7722.993802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7722.997791] Lustre: Skipped 3 previous similar messages [ 7727.303760] Lustre: server umount lustre-MDT0000 complete [ 7728.096792] LustreError: 472457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7728.107347] LustreError: 472457:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7729.062663] LustreError: 472437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778354608 with bad export cookie 12736450815391696603 [ 7729.068789] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7729.070271] LustreError: 472437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7729.214515] Lustre: server umount lustre-MDT0001 complete [ 7733.816615] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 7735.320823] Key type lgssc unregistered [ 7735.486492] LNet: 479906:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7735.492572] LNetError: 479906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7735.507606] LNet: Removed LNI 192.168.204.140@tcp [ 7735.950730] Key type .llcrypt unregistered [ 7735.953053] Key type ._llcrypt unregistered [ 7745.010940] Key type ._llcrypt registered [ 7745.013273] Key type .llcrypt registered [ 7745.075919] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 7745.491147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7745.528857] alg: No test for adler32 (adler32-zlib) [ 7746.452474] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 7746.552251] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 7748.135176] Key type lgssc registered [ 7748.721411] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7766.723775] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 7770.156302] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 7787.034784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7787.046124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7788.192294] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7788.211410] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7788.253240] Lustre: lustre-MDT0000: new disk, initializing [ 7788.293951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7788.304493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7790.172858] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7795.801955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7795.874382] Lustre: 484291:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7795.921366] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7795.925494] Lustre: Skipped 1 previous similar message [ 7795.975559] Lustre: lustre-MDT0001: new disk, initializing [ 7796.034955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7796.057253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7796.064913] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7797.927734] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7801.974177] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7803.784520] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7807.175160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7807.346817] Lustre: lustre-OST0000: new disk, initializing [ 7807.350805] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7807.405417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7809.933809] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7810.769941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7810.776480] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7810.828765] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7814.083320] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7816.102596] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7816.244575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7818.175865] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7818.312223] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7821.287819] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7821.289453] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7821.300419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7821.303932] LustreError: Skipped 1 previous similar message [ 7825.395565] Lustre: server umount lustre-OST0000 complete [ 7831.523431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7831.524444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7831.531065] Lustre: Skipped 2 previous similar messages [ 7831.538187] Lustre: Skipped 2 previous similar messages [ 7833.408396] Lustre: server umount lustre-MDT0000 complete [ 7835.263769] LustreError: 485069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778354714 with bad export cookie 1249345761584125437 [ 7835.269902] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7835.271179] LustreError: 485069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7835.453657] Lustre: server umount lustre-MDT0001 complete [ 7847.400524] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 7852.379594] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 7868.727855] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7589 sec ========== 15:25:46 (1778354746) [ 7869.521720] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:25:47 (1778354747) === [ 7871.181206] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:25:49 (1778354749) === [ 7890.534843] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 7892.170254] Key type lgssc unregistered [ 7892.354835] LNet: 492470:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7892.360204] LNetError: 492470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7892.372883] LNet: Removed LNI 192.168.204.140@tcp [ 7892.859734] Key type .llcrypt unregistered [ 7892.862072] Key type ._llcrypt unregistered