[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 474868255 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a2000-0x1465ccfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002408] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007336] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.009018] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.010010] pid_max: default: 32768 minimum: 301 [ 0.011135] LSM: Security Framework initializing [ 0.013043] Yama: becoming mindful. [ 0.014030] SELinux: Initializing. [ 0.015073] *** VALIDATE selinux *** [ 0.025481] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.030755] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031140] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032096] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034060] *** VALIDATE tmpfs *** [ 0.035621] *** VALIDATE proc *** [ 0.036292] *** VALIDATE cgroup *** [ 0.037008] *** VALIDATE cgroup2 *** [ 0.038209] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039163] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041026] Spectre V2 : User space: Vulnerable [ 0.042005] Speculative Store Bypass: Vulnerable [ 0.045004] debug: unmapping init [mem 0xffffffffa0059000-0xffffffffa0060fff] [ 0.048303] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049738] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050020] ... version: 2 [ 0.051030] ... bit width: 48 [ 0.052007] ... generic registers: 4 [ 0.053006] ... value mask: 0000ffffffffffff [ 0.054010] ... max period: 00007fffffffffff [ 0.055011] ... fixed-purpose events: 3 [ 0.056006] ... event mask: 000000070000000f [ 0.057277] rcu: Hierarchical SRCU implementation. [ 0.059537] smp: Bringing up secondary CPUs ... [ 0.060424] x86: Booting SMP configuration: [ 0.061012] .... node #0, CPUs: #1 #2 #3 [ 0.067223] smp: Brought up 1 node, 4 CPUs [ 0.069012] smpboot: Max logical packages: 1 [ 0.070012] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.090019] node 0 deferred pages initialised in 18ms [ 0.093438] devtmpfs: initialized [ 0.094240] x86/mm: Memory block size: 128MB [ 0.097490] gcov: version magic: 0x41383552 [ 0.099310] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.100072] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.101292] pinctrl core: initialized pinctrl subsystem [ 0.102400] [ 0.102926] ************************************************************* [ 0.103010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.104010] ** ** [ 0.105013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.106014] ** ** [ 0.107015] ** This means that this kernel is built to expose internal ** [ 0.108012] ** IOMMU data structures, which may compromise security on ** [ 0.109012] ** your system. ** [ 0.110011] ** ** [ 0.111013] ** If you see this message and you are not debugging the ** [ 0.112011] ** kernel, report this immediately to your vendor! ** [ 0.113010] ** ** [ 0.114011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.115012] ************************************************************* [ 0.116866] NET: Registered protocol family 16 [ 0.117494] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.118046] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.119117] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.120680] cpuidle: using governor menu [ 0.122556] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.125588] PCI: Using configuration type 1 for base access [ 0.128179] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.136157] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.139016] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.143116] cryptd: max_cpu_qlen set to 1000 [ 0.144461] ACPI: Added _OSI(Module Device) [ 0.145009] ACPI: Added _OSI(Processor Device) [ 0.146000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.149019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.156051] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.163741] ACPI: Interpreter enabled [ 0.164061] ACPI: PM: (supports S0 S3 S4 S5) [ 0.165012] ACPI: Using IOAPIC for interrupt routing [ 0.166245] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.167556] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.178095] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.179178] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.180018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.181080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.183958] acpiphp: Slot [2] registered [ 0.184101] acpiphp: Slot [5] registered [ 0.185219] acpiphp: Slot [6] registered [ 0.186117] acpiphp: Slot [7] registered [ 0.187147] acpiphp: Slot [8] registered [ 0.188347] acpiphp: Slot [9] registered [ 0.189104] acpiphp: Slot [10] registered [ 0.190277] acpiphp: Slot [3] registered [ 0.191286] acpiphp: Slot [4] registered [ 0.192238] acpiphp: Slot [11] registered [ 0.193094] acpiphp: Slot [12] registered [ 0.194141] acpiphp: Slot [13] registered [ 0.195084] acpiphp: Slot [14] registered [ 0.198195] acpiphp: Slot [15] registered [ 0.200124] acpiphp: Slot [16] registered [ 0.203108] acpiphp: Slot [17] registered [ 0.204257] acpiphp: Slot [18] registered [ 0.206105] acpiphp: Slot [19] registered [ 0.208204] acpiphp: Slot [20] registered [ 0.210058] acpiphp: Slot [21] registered [ 0.212102] acpiphp: Slot [22] registered [ 0.214094] acpiphp: Slot [23] registered [ 0.216231] acpiphp: Slot [24] registered [ 0.218303] acpiphp: Slot [25] registered [ 0.221094] acpiphp: Slot [26] registered [ 0.224119] acpiphp: Slot [27] registered [ 0.226342] acpiphp: Slot [28] registered [ 0.229114] acpiphp: Slot [29] registered [ 0.231101] acpiphp: Slot [30] registered [ 0.234122] acpiphp: Slot [31] registered [ 0.236092] PCI host bridge to bus 0000:00 [ 0.239016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.243026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.246022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.252055] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.255029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.259027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.261201] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.264235] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.266133] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.279018] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.285076] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.287019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.289016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.291023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.293554] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.295851] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.299039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.301817] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.306013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.319013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.323011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.333251] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.348024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.357019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.376017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.386000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.393030] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.400015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.421017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.432698] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.441015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.449016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.473017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.490434] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.499015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.506019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.525015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.542105] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.551015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.561030] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.583009] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.594440] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.605021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.612013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.634017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.652131] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.655539] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.658435] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.661368] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.665165] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.675644] iommu: Default domain type: Passthrough [ 0.679451] SCSI subsystem initialized [ 0.681107] ACPI: bus type USB registered [ 0.683124] usbcore: registered new interface driver usbfs [ 0.685149] usbcore: registered new interface driver hub [ 0.687049] usbcore: registered new device driver usb [ 0.688111] pps_core: LinuxPPS API ver. 1 registered [ 0.696015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.698077] PTP clock support registered [ 0.701249] EDAC MC: Ver: 3.0.0 [ 0.704156] PCI: Using ACPI for IRQ routing [ 0.707889] NetLabel: Initializing [ 0.708012] NetLabel: domain hash size = 128 [ 0.710006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.711151] NetLabel: unlabeled traffic allowed by default [ 0.714392] vgaarb: loaded [ 0.716766] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.720016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.727486] clocksource: Switched to clocksource kvm-clock [ 0.854581] VFS: Disk quotas dquot_6.6.0 [ 0.857534] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.861501] *** VALIDATE ramfs *** [ 0.864569] *** VALIDATE hugetlbfs *** [ 0.868984] pnp: PnP ACPI init [ 0.876114] pnp: PnP ACPI: found 6 devices [ 0.899916] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.903535] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.905537] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.907575] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.910778] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.915067] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.919145] NET: Registered protocol family 2 [ 0.921761] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.927392] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.931073] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.939311] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.944861] TCP: Hash tables configured (established 65536 bind 65536) [ 0.955131] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.963056] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.967594] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.971592] NET: Registered protocol family 1 [ 0.974636] RPC: Registered named UNIX socket transport module. [ 0.981497] RPC: Registered udp transport module. [ 0.983665] RPC: Registered tcp transport module. [ 0.985574] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.988375] NET: Registered protocol family 44 [ 0.990027] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.992294] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.994599] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.997125] PCI: CLS 0 bytes, default 64 [ 0.999260] Unpacking initramfs... [ 2.853536] debug: unmapping init [mem 0xffff96803cc54000-0xffff96803ffbffff] [ 2.858591] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.860786] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.863434] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.434150] Initialise system trusted keyrings [ 3.436149] Key type blacklist registered [ 3.438443] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.448634] zbud: loaded [ 3.451705] *** VALIDATE nfs *** [ 3.453044] *** VALIDATE nfs4 *** [ 3.454488] pstore: using deflate compression [ 3.458080] Platform Keyring initialized [ 3.567901] NET: Registered protocol family 38 [ 3.569299] Key type asymmetric registered [ 3.570316] Asymmetric key parser 'x509' registered [ 3.571559] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.574120] io scheduler mq-deadline registered [ 3.575231] io scheduler kyber registered [ 3.576311] io scheduler bfq registered [ 3.577662] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.579885] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.581801] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.588226] ACPI: Power Button [PWRF] [ 3.596531] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.609923] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.633355] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.662305] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.926464] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.964122] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.992907] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.998812] Non-volatile memory driver v1.3 [ 4.000560] Linux agpgart interface v0.103 [ 4.131835] virtio_blk virtio1: [vda] 134776 512-byte logical blocks (69.0 MB/65.8 MiB) [ 4.139077] vda: detected capacity change from 0 to 69005312 [ 4.185796] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.192963] vdb: detected capacity change from 0 to 1073741824 [ 4.234342] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.306038] vdc: detected capacity change from 0 to 2621440000 [ 4.326421] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.330111] vdd: detected capacity change from 0 to 2621440000 [ 4.375787] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.618059] vde: detected capacity change from 0 to 4294967296 [ 4.691461] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.694349] vdf: detected capacity change from 0 to 4294967296 [ 4.702965] libphy: Fixed MDIO Bus: probed [ 4.708211] usbcore: registered new interface driver usbserial_generic [ 4.713481] usbserial: USB Serial support registered for generic [ 4.720721] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.727381] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.731442] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.738260] mousedev: PS/2 mouse device common for all mice [ 4.746806] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.755527] rtc_cmos 00:05: RTC can wake from S4 [ 4.760244] rtc_cmos 00:05: registered as rtc0 [ 4.762892] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.765865] intel_pstate: CPU model not supported [ 4.768503] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.770909] hid: raw HID events driver (C) Jiri Kosina [ 4.776938] usbcore: registered new interface driver usbhid [ 4.779033] usbhid: USB HID core driver [ 4.780514] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.782522] drop_monitor: Initializing network drop monitor service [ 4.787782] Initializing XFRM netlink socket [ 4.789633] NET: Registered protocol family 10 [ 4.792637] Segment Routing with IPv6 [ 4.794179] NET: Registered protocol family 17 [ 4.796042] mpls_gso: MPLS GSO support [ 4.806435] RAS: Correctable Errors collector initialized. [ 4.809044] AVX version of gcm_enc/dec engaged. [ 4.812326] AES CTR mode by8 optimization enabled [ 4.990261] sched_clock: Marking stable (4990022318, 0)->(6031650520, -1041628202) [ 5.005163] registered taskstats version 1 [ 5.006977] Loading compiled-in X.509 certificates [ 5.019526] zswap: loaded using pool lzo/zbud [ 5.052529] Key type big_key registered [ 5.065524] Key type encrypted registered [ 5.066762] ima: No TPM chip found, activating TPM-bypass! [ 5.068291] ima: Allocated hash algorithm: sha1 [ 5.069782] ima: No architecture policies found [ 5.071632] evm: Initialising EVM extended attributes: [ 5.073759] evm: security.selinux [ 5.075091] evm: security.ima [ 5.076242] evm: security.capability [ 5.077455] evm: HMAC attrs: 0x1 [ 5.080275] rtc_cmos 00:05: setting system clock to 2026-05-15 09:55:56 UTC (1778838956) [ 5.087929] debug: unmapping init [mem 0xffffffffa1003000-0xffffffffa11fffff] [ 5.090636] debug: unmapping init [mem 0xffffffff9fd82000-0xffffffffa0058fff] [ 5.095051] Write protecting the kernel read-only data: 28672k [ 5.100096] debug: unmapping init [mem 0xffffffff9e403000-0xffffffff9e5fffff] [ 5.102736] debug: unmapping init [mem 0xffffffff9ed14000-0xffffffff9edfffff] [ 5.151426] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.164622] systemd[1]: Detected virtualization kvm. [ 5.168297] systemd[1]: Detected architecture x86-64. [ 5.170611] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.202254] systemd[1]: No hostname configured. [ 5.206151] systemd[1]: Set hostname to . [ 5.208436] random: systemd: uninitialized urandom read (16 bytes read) [ 5.211130] systemd[1]: Initializing machine ID from random generator. [ 5.351717] random: ln: uninitialized urandom read (6 bytes read) [ 5.578946] random: systemd: uninitialized urandom read (16 bytes read) [ 5.583673] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 5.608798] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 5.631410] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.115783] device-mapper: uevent: version 1.0.3 [ 7.118928] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [[ 8.193330] random: fast init done  OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.573542] virtio_net virtio0 ens2: renamed from eth0 [ 8.869158] scsi host0: ata_piix [ 9.264847] scsi host1: ata_piix [ 9.272027] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.283040] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.654682] random: crng init done [ 15.659942] random: 7 urandom warning(s) missed due to ratelimiting [ 16.890515] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 18.907837] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 21.296201] printk: systemd: 25 output lines suppressed due to ratelimiting [ 21.801081] SELinux: Disabled at runtime. [ 21.880471] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.894726] systemd[1]: Detected virtualization kvm. [ 21.897136] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 23.337411] systemd[1]: initrd-switch-root.service: Succeeded. [ 23.358821] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 23.378603] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 23.383307] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 23.387290] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 23.408605] systemd[1]: Starting Journal Service... Starting Journal Service... [ 23.426073] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK [ 23.593320] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 24.868881] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 25.998093] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.051906] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 27.095411] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 27.485148] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ ***] A start job is running for Configur…-only root support (9s / no limit) [ **] A start job is running for Configur…only root support (10s / no limit)[ 33.844572] Key type dns_resolver registered [ *] A start job is running for Configur…only root support (10s / no limit)[ 34.418746] NFS: Registering the id_resolver key type [ 34.420855] Key type id_resolver registered [ 34.423149] Key type id_legacy registered [ **] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg426-server login: [ 64.560076] libcfs: loading out-of-tree module taints kernel. [ 64.576722] Key type ._llcrypt registered [ 64.578177] Key type .llcrypt registered [ 64.620784] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 72.531387] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 73.089722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 73.095336] alg: No test for adler32 (adler32-zlib) [ 74.047206] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 74.307028] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 75.911155] Key type lgssc registered [ 76.554482] Lustre: Echo OBD driver; http://www.lustre.org/ [ 83.491101] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 99.970974] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 105.007497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 105.016907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 106.151633] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 106.173850] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 106.220249] Lustre: lustre-MDT0000: new disk, initializing [ 106.259958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 106.269838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 107.835176] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 113.812985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 113.858651] Lustre: 6482:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 113.876219] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 113.877638] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 113.882988] Lustre: Skipped 1 previous similar message [ 113.930447] Lustre: lustre-MDT0001: new disk, initializing [ 113.962209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 113.978257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 113.984422] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 115.489948] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 118.035536] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 121.826528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 121.924821] Lustre: lustre-OST0000: new disk, initializing [ 121.926833] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 121.929527] Lustre: 8388:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 121.952213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 123.896351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 123.900359] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 123.929376] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 123.962458] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 130.869990] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 130.923043] Lustre: lustre-OST0001: new disk, initializing [ 130.925878] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 130.929359] Lustre: 9446:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 130.958441] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 133.430435] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 136.173842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 136.177202] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 136.197257] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 139.638278] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 144.473720] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 146.633632] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing check_logdir /tmp/testlogs/ [ 148.238087] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing yml_node [ 149.731766] Lustre: DEBUG MARKER: Client: 2.17.53.3 [ 150.657146] Lustre: DEBUG MARKER: MDS: 2.17.53.3 [ 151.714377] Lustre: DEBUG MARKER: OSS: 2.17.53.3 [ 152.462482] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri May 15 05:58:22 EDT 2026 [ 159.255722] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 159.873854] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 172.000088] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 172.000533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 172.006586] Lustre: Skipped 2 previous similar messages [ 172.010506] Lustre: Skipped 1 previous similar message [ 177.120762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 177.123808] Lustre: Skipped 3 previous similar messages [ 182.240829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 182.244671] Lustre: Skipped 3 previous similar messages [ 186.335723] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 186.408591] Lustre: server umount lustre-MDT0000 complete [ 187.932942] LustreError: 6475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778839139 with bad export cookie 9610534775357084774 [ 187.934205] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 187.939673] LustreError: 6475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 192.480037] LustreError: 6491:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 192.480261] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 192.488177] LustreError: 6491:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 6 previous similar messages [ 192.492883] Lustre: Skipped 1 previous similar message [ 192.494492] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 192.502283] Lustre: Skipped 1 previous similar message [ 197.601442] LustreError: 6519:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 197.610942] LustreError: 6519:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 202.207161] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 202.305431] Lustre: server umount lustre-MDT0001 complete [ 210.116279] Lustre: server umount lustre-OST0000 complete [ 217.753216] Lustre: server umount lustre-OST0001 complete [ 220.303143] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 223.964981] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 238.684467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 238.776757] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 238.791601] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 238.829926] Lustre: lustre-MDT0000: new disk, initializing [ 238.858088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 238.865555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 240.233503] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 244.729422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 244.778503] Lustre: 14950:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 244.918673] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 246.477381] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 250.265939] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 251.780757] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 254.584976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 254.707902] Lustre: lustre-OST0000: new disk, initializing [ 254.709868] Lustre: Skipped 1 previous similar message [ 254.711892] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 254.714082] Lustre: Skipped 2 previous similar messages [ 254.716850] Lustre: 16690:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 254.758576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 254.761684] Lustre: Skipped 1 previous similar message [ 255.807900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 255.812525] Lustre: Skipped 1 previous similar message [ 255.815051] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 255.854591] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 257.000639] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 260.608575] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 262.277340] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 262.368173] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 264.040264] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 264.140655] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 266.211904] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 266.212189] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 266.215144] LustreError: Skipped 1 previous similar message [ 266.220607] Lustre: Skipped 1 previous similar message [ 266.226928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 266.230198] Lustre: Skipped 3 previous similar messages [ 271.005588] Lustre: server umount lustre-OST0000 complete [ 275.936131] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 275.939632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 278.810670] Lustre: server umount lustre-MDT0000 complete [ 280.226693] LustreError: 14944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778839231 with bad export cookie 9610534775357086279 [ 280.232228] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 280.234611] LustreError: 14944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 280.359695] Lustre: server umount lustre-MDT0001 complete [ 281.945037] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 06:00:32 (1778839232) [ 282.898758] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 283.554607] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 06:00:33 (1778839233) [ 286.041051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 286.249228] LustreError: 18949:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 286.277255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 287.817461] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 291.154641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 292.867822] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 295.000708] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 296.491795] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 299.639176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 302.395791] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 305.565249] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 321.504905] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 321.506541] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 321.508622] LustreError: Skipped 1 previous similar message [ 321.514720] Lustre: Skipped 1 previous similar message [ 321.515408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 321.521580] Lustre: Skipped 4 previous similar messages [ 325.851900] Lustre: server umount lustre-OST0000 complete [ 331.744740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 331.749587] Lustre: Skipped 1 previous similar message [ 333.629586] Lustre: server umount lustre-MDT0000 complete [ 335.423871] LustreError: 18929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778839286 with bad export cookie 9610534775357087161 [ 335.426913] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 335.430638] LustreError: 18929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 335.597947] Lustre: server umount lustre-MDT0001 complete [ 340.788768] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 342.207926] Key type lgssc unregistered [ 342.367768] LNet: 22159:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 342.376741] LNetError: 22159:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 342.393215] LNet: Removed LNI 192.168.204.126@tcp [ 342.831131] Key type .llcrypt unregistered [ 342.832368] Key type ._llcrypt unregistered [ 352.972500] Key type ._llcrypt registered [ 352.974073] Key type .llcrypt registered [ 353.025474] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 353.435980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 353.449929] alg: No test for adler32 (adler32-zlib) [ 354.331659] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 354.431539] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 356.023174] Key type lgssc registered [ 356.434298] Lustre: Echo OBD driver; http://www.lustre.org/ [ 359.560967] loop: module loaded [ 359.580746] loop0: detected capacity change from 0 to 204800000 [ 359.694689] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 359.699831] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 359.828726] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 362.054441] loop0: detected capacity change from 0 to 204800000 [ 362.118365] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 362.123643] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 362.130067] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 364.051857] loop0: detected capacity change from 0 to 204800000 [ 366.321841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 366.329764] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 367.446406] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 367.466375] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 367.508596] Lustre: lustre-MDT0000: new disk, initializing [ 367.546109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 367.558804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 369.133825] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 373.575811] loop1: detected capacity change from 0 to 204800000 [ 376.129595] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 376.268146] Lustre: lustre-OST07c6: new disk, initializing [ 376.270766] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 376.273013] Lustre: Skipped 1 previous similar message [ 376.275872] Lustre: 25287:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 376.310663] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 379.039508] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 380.458257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 380.464890] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 380.483621] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 381.766759] Lustre: 24290:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 381.771780] Lustre: 24290:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 381.776658] Lustre: 24290:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 381.780287] Lustre: 24290:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 381.784150] Lustre: 24290:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 381.787384] Lustre: 24290:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 398.421090] Lustre: 24291:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 398.425866] Lustre: 24291:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 398.429676] Lustre: 24291:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 398.435923] Lustre: 24291:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 398.441915] Lustre: 24291:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 398.446602] Lustre: 24291:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 402.212477] Lustre: 25763:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 402.217909] Lustre: 25763:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 402.221871] Lustre: 25763:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 402.225859] Lustre: 25763:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 402.229383] Lustre: 25763:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 402.233143] Lustre: 25763:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 402.237176] Lustre: 25763:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 402.241168] Lustre: 25763:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 402.245209] Lustre: 25763:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 402.249009] Lustre: 25763:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 402.253445] Lustre: 25763:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 402.257154] Lustre: 25763:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 405.333826] Lustre: 24290:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 405.337717] Lustre: 24290:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 405.340659] Lustre: 24290:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 405.343021] Lustre: 24290:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 405.346466] Lustre: 24290:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 405.349684] Lustre: 24290:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 405.353829] Lustre: 24290:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 405.357578] Lustre: 24290:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 405.360871] Lustre: 24290:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 405.363847] Lustre: 24290:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 405.368247] Lustre: 24290:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 405.371663] Lustre: 24290:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 412.196558] Lustre: 24289:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 412.200618] Lustre: 24289:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 412.205134] Lustre: 24289:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 412.210466] Lustre: 24289:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 412.216201] Lustre: 24289:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 412.223492] Lustre: 24289:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 412.227402] Lustre: 24289:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 412.231298] Lustre: 24289:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 412.235547] Lustre: 24289:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 412.240377] Lustre: 24289:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 412.245874] Lustre: 24289:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 412.250256] Lustre: 24289:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 422.129451] Lustre: 25763:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 422.134240] Lustre: 25763:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 422.137442] Lustre: 25763:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 422.141387] Lustre: 25763:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 422.144644] Lustre: 25763:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 422.148534] Lustre: 25763:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 422.153211] Lustre: 25763:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 422.158529] Lustre: 25763:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 422.163519] Lustre: 25763:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 422.168106] Lustre: 25763:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 422.172450] Lustre: 25763:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 422.177030] Lustre: 25763:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 467.428229] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 467.438657] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 467.439175] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 469.934608] Lustre: server umount lustre-OST07c6 complete [ 474.059146] Lustre: server umount lustre-MDT0000 complete [ 481.578668] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 482.879959] Key type lgssc unregistered [ 483.041153] LNet: 27832:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 483.046169] LNetError: 27832:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 483.059165] LNet: Removed LNI 192.168.204.126@tcp [ 483.413548] Key type .llcrypt unregistered [ 483.415213] Key type ._llcrypt unregistered [ 492.282911] Key type ._llcrypt registered [ 492.285517] Key type .llcrypt registered [ 492.337882] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 492.745495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 492.781270] alg: No test for adler32 (adler32-zlib) [ 493.718694] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 493.820105] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 495.415210] Key type lgssc registered [ 495.868242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 499.179409] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 499.189039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 500.408674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 501.993186] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 505.578545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 505.729471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 507.340641] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 509.667809] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 511.170913] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 513.869115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 514.008949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 516.236316] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 518.474571] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 525.288351] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 536.544835] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 536.548215] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 536.554975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 538.082252] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 538.082281] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 538.087610] LustreError: Skipped 1 previous similar message [ 538.094154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 540.641821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 540.646428] Lustre: Skipped 1 previous similar message [ 541.727292] Lustre: server umount lustre-OST0000 complete [ 545.761202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 545.762333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 545.767111] Lustre: Skipped 1 previous similar message [ 545.773577] Lustre: Skipped 1 previous similar message [ 549.416196] Lustre: server umount lustre-MDT0000 complete [ 550.880267] LustreError: 30956:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.887738] LustreError: 30956:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 551.013700] LustreError: 29043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778839502 with bad export cookie 4552322423039206634 [ 551.017154] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 551.021336] LustreError: 29043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 551.151244] Lustre: server umount lustre-MDT0001 complete [ 555.464641] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 556.681342] Key type lgssc unregistered [ 556.831293] LNet: 32459:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 556.835059] LNetError: 32459:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 556.850409] LNet: Removed LNI 192.168.204.126@tcp [ 557.249709] Key type .llcrypt unregistered [ 557.251602] Key type ._llcrypt unregistered [ 565.659378] Key type ._llcrypt registered [ 565.661924] Key type .llcrypt registered [ 565.706881] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 566.114782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 566.139091] alg: No test for adler32 (adler32-zlib) [ 567.020200] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 567.117204] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 568.711824] Key type lgssc registered [ 569.182291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 572.181558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 572.190725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 573.406108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 575.057567] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 578.493634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 578.665419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 580.256960] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 582.407920] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 583.741811] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 586.493240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 586.634692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 589.003368] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 591.362217] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 597.926317] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 608.228617] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 608.228648] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 608.229322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 608.234678] Lustre: Skipped 1 previous similar message [ 611.933423] Lustre: server umount lustre-OST0000 complete [ 614.368033] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 614.371799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 614.377988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 614.380327] Lustre: Skipped 1 previous similar message [ 618.464276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 618.466234] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 619.556940] Lustre: server umount lustre-MDT0000 complete [ 621.023318] LustreError: 33668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778839572 with bad export cookie 6792196817297331915 [ 621.025433] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 621.028525] LustreError: 33668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 621.156728] Lustre: server umount lustre-MDT0001 complete [ 625.355707] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 626.563674] Key type lgssc unregistered [ 626.702560] LNet: 36883:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 626.706407] LNetError: 36883:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 626.714299] LNet: Removed LNI 192.168.204.126@tcp [ 627.070099] Key type .llcrypt unregistered [ 627.071944] Key type ._llcrypt unregistered [ 637.062048] Key type ._llcrypt registered [ 637.063524] Key type .llcrypt registered [ 637.105028] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 637.462899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 637.478183] alg: No test for adler32 (adler32-zlib) [ 638.350915] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 638.439496] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 640.031249] Key type lgssc registered [ 640.497709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 643.658567] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 643.666990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 644.864723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 646.481759] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 650.138202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 650.308336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 652.012064] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 654.301887] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 655.838648] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 658.674292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 658.834937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 661.134364] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 663.598357] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 671.148768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 676.669426] Lustre: server umount lustre-MDT0000 complete [ 678.216427] LustreError: 39877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778839629 with bad export cookie 2631640100016071585 [ 678.220461] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 678.222086] LustreError: 39877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 679.121806] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.26@tcp (stopping) [ 679.132448] LustreError: 40411:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 679.140202] LustreError: 40411:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 681.441164] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 681.449722] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 681.449759] LustreError: 38258:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 683.488370] LustreError: 40411:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 683.488380] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 684.409317] Lustre: server umount lustre-MDT0001 complete [ 702.559646] Lustre: server umount lustre-OST0000 complete [ 709.810595] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 711.088155] Key type lgssc unregistered [ 711.246567] LNet: 41779:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 711.251739] LNetError: 41779:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 711.262687] LNet: Removed LNI 192.168.204.126@tcp [ 711.649936] Key type .llcrypt unregistered [ 711.652249] Key type ._llcrypt unregistered [ 722.129315] Key type ._llcrypt registered [ 722.131012] Key type .llcrypt registered [ 722.183825] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 722.540267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 722.585169] alg: No test for adler32 (adler32-zlib) [ 723.479683] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 723.585176] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 725.175765] Key type lgssc registered [ 725.637508] Lustre: Echo OBD driver; http://www.lustre.org/ [ 729.103468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 729.112876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 730.329836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 732.022956] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 735.958894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 736.137514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 737.867139] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 740.140720] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 741.577438] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 744.346440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 744.509816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 747.046403] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 749.300673] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 754.789937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 760.289631] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 760.290642] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 760.294402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 760.298854] Lustre: Skipped 1 previous similar message [ 761.052890] Lustre: lustre-OST0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 761.057083] Lustre: Skipped 1 previous similar message [ 764.388486] Lustre: server umount lustre-OST0000 complete [ 779.809926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 779.813596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 779.816660] Lustre: Skipped 1 previous similar message [ 783.967530] Lustre: server umount lustre-MDT0000 complete [ 784.864165] LustreError: 43156:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 784.873238] LustreError: 43156:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 785.434309] LustreError: 44770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778839736 with bad export cookie 17913936420685798740 [ 785.438147] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 785.440207] LustreError: 44770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 791.630957] Lustre: server umount lustre-MDT0001 complete [ 795.833343] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 797.067627] Key type lgssc unregistered [ 797.215607] LNet: 46505:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 797.219033] LNetError: 46505:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 797.225383] LNet: Removed LNI 192.168.204.126@tcp [ 797.580587] Key type .llcrypt unregistered [ 797.582375] Key type ._llcrypt unregistered [ 808.044839] Key type ._llcrypt registered [ 808.046296] Key type .llcrypt registered [ 808.094775] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 808.559372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 808.566028] alg: No test for adler32 (adler32-zlib) [ 809.446391] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 809.544703] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 811.143159] Key type lgssc registered [ 811.616086] Lustre: Echo OBD driver; http://www.lustre.org/ [ 814.866794] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 814.878721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 816.109771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 817.785896] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 821.458325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 821.636623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 823.325872] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 825.445492] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 826.899359] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 829.579232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 829.696980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 831.707987] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 833.710448] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 840.998180] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 846.616920] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 847.771893] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 858.533132] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-05-15 6h10m08s [ 859.029687] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 859.862153] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 860.874950] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 861.664450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 861.669503] Lustre: Skipped 2 previous similar messages [ 862.688921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 862.691742] Lustre: Skipped 5 previous similar messages [ 865.880740] Lustre: server umount lustre-MDT0000 complete [ 867.020151] LustreError: 47878:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 867.026162] LustreError: 47878:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 867.064300] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 868.832592] LustreError: 49044:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 868.839574] LustreError: 49044:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 869.275798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 869.323095] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 869.411360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 869.434420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 870.438538] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 870.739521] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 871.837343] Lustre: DEBUG MARKER: Wait for df (35932) ... [ 872.326304] Lustre: DEBUG MARKER: done [ 874.464144] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 874.466725] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 874.470461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 875.344199] Lustre: server umount lustre-OST0000 complete [ 878.670861] Lustre: server umount lustre-MDT0000 complete [ 879.912241] LustreError: 47862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778839831 with bad export cookie 5741485926270450923 [ 879.915652] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 879.916997] LustreError: 47862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 890.389232] Lustre: server umount lustre-MDT0001 complete [ 894.142475] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 895.242271] Key type lgssc unregistered [ 895.375411] LNet: 52443:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 895.379050] LNetError: 52443:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 895.385405] LNet: Removed LNI 192.168.204.126@tcp [ 895.701905] Key type .llcrypt unregistered [ 895.703431] Key type ._llcrypt unregistered [ 911.116483] Key type ._llcrypt registered [ 911.118033] Key type .llcrypt registered [ 911.164273] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 911.509162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 911.552044] alg: No test for adler32 (adler32-zlib) [ 912.409943] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 912.504777] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 914.087165] Key type lgssc registered [ 914.486756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 917.134119] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 917.142544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 918.240918] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 918.252879] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 918.319067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 919.675628] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 922.623544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 922.668413] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 922.679457] Lustre: 54613:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 922.774491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 924.150693] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 926.018906] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 927.234877] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 929.491096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 929.561496] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 929.624933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 931.623559] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 933.590590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 933.659572] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 937.075910] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-05-15 6h11m27s [ 937.593832] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 939.990740] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 940.825056] Lustre: DEBUG MARKER: Creating a test file and stat it [ 940.840201] LustreError: 54619:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 961.244549] Lustre: lustre-MDT0000: Client a6608e31-3592-4120-8998-a3bd33095e13 (at 192.168.204.26@tcp) reconnecting [ 980.887117] LustreError: 54619:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 981.648670] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 982.539967] Lustre: DEBUG MARKER: done [ 985.056507] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 985.058100] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 985.060525] LustreError: Skipped 1 previous similar message [ 985.067604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 985.072287] Lustre: Skipped 1 previous similar message [ 990.177436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 990.180997] Lustre: Skipped 1 previous similar message [ 990.294764] Lustre: server umount lustre-OST0000 complete [ 994.784279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 994.787962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 994.792297] Lustre: Skipped 1 previous similar message [ 994.794699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 997.789085] Lustre: server umount lustre-MDT0000 complete [ 999.064439] LustreError: 55337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778839950 with bad export cookie 2010450392773459384 [ 999.068812] LustreError: 55337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 999.069503] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 999.200596] Lustre: server umount lustre-MDT0001 complete [ 1002.997392] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1004.125495] Key type lgssc unregistered [ 1004.270576] LNet: 58578:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1004.274485] LNetError: 58578:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1004.283434] LNet: Removed LNI 192.168.204.126@tcp [ 1004.601523] Key type .llcrypt unregistered [ 1004.603256] Key type ._llcrypt unregistered [ 1021.081099] Key type ._llcrypt registered [ 1021.082422] Key type .llcrypt registered [ 1021.120866] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1021.556633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1021.566654] alg: No test for adler32 (adler32-zlib) [ 1022.420671] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 1022.506290] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1024.095322] Key type lgssc registered [ 1024.497154] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1025.718082] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1025.725290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1026.836220] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1026.848647] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1026.923597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1027.290574] Lustre: Failing over lustre-MDT0000 [ 1027.397572] Lustre: server umount lustre-MDT0000 complete [ 1029.803879] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 06:13:00 (1778839980) [ 1032.340681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1032.522251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1033.927579] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1037.000190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1037.042859] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1037.055965] Lustre: 61335:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1037.172225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1038.622128] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1040.732862] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1042.026846] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1044.481534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1044.561940] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1044.631245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1045.047538] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1046.645081] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1048.679960] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1053.048609] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1055.201120] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1055.202699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1055.206513] Lustre: Skipped 2 previous similar messages [ 1055.210843] Lustre: Skipped 1 previous similar message [ 1060.320845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1060.323821] Lustre: Skipped 3 previous similar messages [ 1061.277826] Lustre: server umount lustre-MDT0000 complete [ 1062.678306] LustreError: 61326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778840014 with bad export cookie 4618654643921110332 [ 1062.680757] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1062.683631] LustreError: 61326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1062.802808] Lustre: server umount lustre-MDT0001 complete [ 1064.110294] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1065.891049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1068.977900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1073.567979] LustreError: 60257:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000038250000 x1865248903560320/t0(0) o250->MGC192.168.204.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1073.720286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1073.752405] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1075.137154] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1078.104627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1078.202332] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1079.642441] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1081.311109] Lustre: 60260:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778840016/real 1778840016] req@00000000c13b98cd x1865248903559936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778840032 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1081.558585] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1082.837155] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1083.359239] Lustre: 60261:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778840018/real 1778840018] req@00000000e55b8bd1 x1865248903560064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778840034 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1083.368937] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1085.413137] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1086.431100] Lustre: 60258:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778840021/real 1778840021] req@000000008250c5bc x1865248903560192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778840037 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1089.503067] Lustre: 60259:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778840024/real 1778840024] req@00000000719fc768 x1865248903560448/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778840040 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1090.530564] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1090.531220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1090.534793] Lustre: Skipped 1 previous similar message [ 1095.648690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1095.651151] Lustre: Skipped 4 previous similar messages [ 1096.029676] Lustre: server umount lustre-MDT0000 complete [ 1097.357327] LustreError: 63569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778840048 with bad export cookie 4618654643921112600 [ 1097.362493] LustreError: 63569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1097.362670] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1097.473317] Lustre: server umount lustre-MDT0001 complete [ 1099.623332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1102.803756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1106.911675] LustreError: 60257:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a06a734f x1865248903581056/t0(0) o250->MGC192.168.204.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1107.042447] LustreError: 62956:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1107.043739] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1107.049725] LustreError: 62956:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1107.054964] Lustre: Skipped 1 previous similar message [ 1107.080764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1107.083173] Lustre: Skipped 1 previous similar message [ 1107.110443] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1108.645926] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1111.798354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1113.419826] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1115.383515] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1116.629424] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1117.023115] Lustre: 60261:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778840052/real 1778840052] req@0000000073ded960 x1865248903580672/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778840068 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1117.161237] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1121.250635] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1126.368590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1126.370224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1126.372567] Lustre: Skipped 2 previous similar messages [ 1126.374553] Lustre: Skipped 1 previous similar message [ 1129.815084] Lustre: server umount lustre-MDT0000 complete [ 1131.119228] LustreError: 62053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778840082 with bad export cookie 4618654643921113979 [ 1131.122325] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1131.123808] LustreError: 62053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1131.241729] Lustre: server umount lustre-MDT0001 complete [ 1133.467387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1134.759373] Lustre: DEBUG MARKER: files compared the same [ 1144.740461] Lustre: server umount lustre-OST0000 complete [ 1150.355311] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1151.386465] Key type lgssc unregistered [ 1151.517378] LNet: 70239:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1151.520352] LNetError: 70239:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1151.530296] LNet: Removed LNI 192.168.204.126@tcp [ 1151.806114] Key type .llcrypt unregistered [ 1151.807383] Key type ._llcrypt unregistered [ 1160.706369] Key type ._llcrypt registered [ 1160.707830] Key type .llcrypt registered [ 1160.754681] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1161.131244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1161.169552] alg: No test for adler32 (adler32-zlib) [ 1162.021917] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 1162.112815] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1163.703180] Key type lgssc registered [ 1164.106136] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1167.064332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1167.072217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1168.296469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1169.698376] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug +malloc all [ 1172.691082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1172.796058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1174.083105] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug +malloc all [ 1175.922383] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1177.102227] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1179.318627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1179.432228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1181.365207] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug +malloc all [ 1183.269118] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1192.741494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1198.047982] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1198.048361] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1198.048693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1198.059308] Lustre: Skipped 1 previous similar message [ 1202.392596] Lustre: server umount lustre-OST0000 complete [ 1208.287754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1208.293432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1208.296335] Lustre: Skipped 2 previous similar messages [ 1210.003496] Lustre: server umount lustre-MDT0000 complete [ 1211.259675] LustreError: 71641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778840162 with bad export cookie 973630572332022927 [ 1211.261052] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1211.266963] LustreError: 71641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1211.376306] Lustre: server umount lustre-MDT0001 complete [ 1215.125237] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1216.151875] Key type lgssc unregistered [ 1216.280363] LNet: 74820:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1216.283185] LNetError: 74820:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1216.292403] LNet: Removed LNI 192.168.204.126@tcp [ 1216.571131] Key type .llcrypt unregistered [ 1216.572426] Key type ._llcrypt unregistered [ 1225.246761] Key type ._llcrypt registered [ 1225.247745] Key type .llcrypt registered [ 1225.284336] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1225.662817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1225.695420] alg: No test for adler32 (adler32-zlib) [ 1226.561314] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 1226.646376] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1228.231125] Key type lgssc registered [ 1228.576036] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1231.193302] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1231.200428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1246.623258] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1251.743224] LustreError: 76169:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.126@tcp: failed processing log, type 4: rc = -110 [ 1278.367197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1280.401913] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1282.331355] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1284.988450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1285.030329] Lustre: *** cfs_fail_loc=706, val=0*** [ 1285.156417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1285.181548] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1286.563989] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1289.745938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1289.887782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1291.302385] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1293.221594] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1294.491394] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1295.328775] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1295.328877] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1295.331655] LustreError: Skipped 1 previous similar message [ 1295.337153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1295.340499] Lustre: Skipped 1 previous similar message [ 1298.401655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1298.404602] Lustre: Skipped 1 previous similar message [ 1301.327246] Lustre: server umount lustre-OST0000 complete [ 1303.520301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1303.521679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1303.524922] Lustre: Skipped 2 previous similar messages [ 1308.639976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1308.641958] Lustre: Skipped 2 previous similar messages [ 1308.759376] Lustre: server umount lustre-MDT0000 complete [ 1310.107203] LustreError: 76176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778840261 with bad export cookie 14905794681280854414 [ 1310.110283] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1310.113357] LustreError: 76176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1310.229640] Lustre: server umount lustre-MDT0001 complete [ 1313.984622] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1315.012404] Key type lgssc unregistered [ 1315.132331] LNet: 79425:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1315.135589] LNetError: 79425:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1315.144321] LNet: Removed LNI 192.168.204.126@tcp [ 1315.452888] Key type .llcrypt unregistered [ 1315.454533] Key type ._llcrypt unregistered [ 1324.449168] Key type ._llcrypt registered [ 1324.450667] Key type .llcrypt registered [ 1324.494141] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1324.833516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1324.870973] alg: No test for adler32 (adler32-zlib) [ 1325.726656] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 1325.816501] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1327.407151] Key type lgssc registered [ 1327.772353] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1330.317987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1330.323715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1332.657420] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1335.221052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1335.424041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1336.701248] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1339.470028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1339.578651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1341.523168] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1344.250053] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1344.256049] Lustre: MGS: Client a8ef3d92-0a0f-4815-85ec-e46b8f9310e5 (at 0@lo) reconnecting [ 1344.347163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1344.383332] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1345.543568] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1354.719956] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1354.722611] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1354.726827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1355.745588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1355.746641] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1355.746778] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1359.313752] Lustre: server umount lustre-OST0000 complete [ 1360.865304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1360.865368] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1360.870337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1360.875170] Lustre: Skipped 2 previous similar messages [ 1365.983986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1365.986694] Lustre: Skipped 1 previous similar message [ 1366.657758] Lustre: server umount lustre-MDT0000 complete [ 1368.036677] Lustre: server umount lustre-MDT0001 complete [ 1369.303467] Lustre: server umount MGS complete [ 1372.989257] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 06:18:43 (1778840323) [ 1382.390941] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 1385.514944] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1397.475162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1398.877551] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1402.903032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1402.990581] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1403.010565] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1403.049227] Lustre: lustre-MDT0001: new disk, initializing [ 1403.071935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1403.081180] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1403.083829] Lustre: 88526:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1403.087069] LustreError: 88526:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1404.427509] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1408.696681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1408.790839] Lustre: lustre-OST0000: new disk, initializing [ 1408.792424] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1408.794816] Lustre: 89275:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1408.813610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1410.810936] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1413.947764] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1414.623710] Lustre: 89823:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000008250c5bc x1865249222329216/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1778840382 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1416.258697] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1416.264872] Lustre: MGS: Client 4738e632-e719-4a8f-82f7-9b44e62b5c2c (at 0@lo) reconnecting [ 1416.275959] LustreError: 88510:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1416.288247] Lustre: 87811:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1416.305310] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1416.341459] Lustre: lustre-MDT0000: new disk, initializing [ 1416.395362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1417.439840] Lustre: 90229:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000bdbd8d3e x1865249222334720/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1778840384 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1417.503483] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1417.738424] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1422.499341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1422.503852] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1422.567544] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1442.276257] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1442.277355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1442.280205] Lustre: Skipped 2 previous similar messages [ 1442.284985] Lustre: Skipped 1 previous similar message [ 1447.695966] Lustre: server umount lustre-OST0000 complete [ 1452.512354] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1452.512739] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1452.515747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1452.521804] Lustre: Skipped 1 previous similar message [ 1452.523663] Lustre: Skipped 2 previous similar messages [ 1463.263083] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1463.303039] Lustre: server umount lustre-MDT0000 complete [ 1470.694664] Lustre: server umount lustre-MDT0001 complete [ 1472.051777] Lustre: server umount MGS complete [ 1475.857324] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 06:20:26 (1778840426) [ 1481.397639] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1482.416319] Key type lgssc unregistered [ 1482.543413] LNet: 92690:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1482.546325] LNetError: 92690:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1482.554493] LNet: Removed LNI 192.168.204.126@tcp [ 1482.840348] Key type .llcrypt unregistered [ 1482.841587] Key type ._llcrypt unregistered [ 1489.535637] Key type ._llcrypt registered [ 1489.536702] Key type .llcrypt registered [ 1489.571533] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1489.854127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1489.861617] alg: No test for adler32 (adler32-zlib) [ 1490.704379] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 1490.776033] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1492.351153] Key type lgssc registered [ 1492.705133] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1495.084559] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1495.092626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1496.232164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1497.388976] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1499.922748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1500.035054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1501.201967] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1502.853107] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1503.981404] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1506.122125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1506.233639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1507.236421] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 1507.994316] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1509.786713] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1515.490854] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1515.491037] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1515.498533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1515.502649] Lustre: Skipped 1 previous similar message [ 1519.303338] Lustre: server umount lustre-OST0000 complete [ 1520.608373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1520.608473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1520.609033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1520.615647] Lustre: Skipped 2 previous similar messages [ 1525.727929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1525.729768] Lustre: Skipped 2 previous similar messages [ 1526.542361] Lustre: server umount lustre-MDT0000 complete [ 1527.691129] LustreError: 93855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778840479 with bad export cookie 12199748511786375800 [ 1527.694159] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1527.696382] LustreError: 93855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1527.797761] Lustre: server umount lustre-MDT0001 complete [ 1531.120741] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1532.152892] Key type lgssc unregistered [ 1532.288374] LNet: 97043:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1532.290890] LNetError: 97043:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1532.297292] LNet: Removed LNI 192.168.204.126@tcp [ 1532.574290] Key type .llcrypt unregistered [ 1532.575770] Key type ._llcrypt unregistered [ 1539.152455] Key type ._llcrypt registered [ 1539.153652] Key type .llcrypt registered [ 1539.189242] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1539.545640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1539.597821] alg: No test for adler32 (adler32-zlib) [ 1540.461404] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 1540.543976] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1542.119150] Key type lgssc registered [ 1542.487538] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1544.542070] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1544.542160] LustreError: 98086:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 1544.545031] Lustre: Skipped 1 previous similar message [ 1544.548965] LustreError: 98087:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 1544.551753] LustreError: 98086:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 1544.554283] LustreError: 98086:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1544.557692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1544.559563] LustreError: 98086:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1545.713357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1548.128067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1548.240662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1549.502705] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1550.970324] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing lsmod [ 1551.895936] LustreError: 99259:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 1551.896164] LustreError: 99258:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 1551.902385] LustreError: 99259:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 1551.905470] LustreError: 99259:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1551.908264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1551.909584] LustreError: 99259:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1552.020309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1553.375854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1553.378671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1553.383237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1557.472232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1557.473197] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1557.473962] Lustre: Skipped 1 previous similar message [ 1557.478188] Lustre: Skipped 1 previous similar message [ 1558.498165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1558.502387] Lustre: Skipped 3 previous similar messages [ 1563.616247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1563.618455] Lustre: Skipped 2 previous similar messages [ 1567.199110] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1567.253217] Lustre: server umount lustre-MDT0000 complete [ 1568.456239] LustreError: 98101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778840519 with bad export cookie 2033405281832938429 [ 1568.458244] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1568.460629] LustreError: 98101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1568.695212] LustreError: 98555:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c58fee3a x1865249446707328/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1568.747272] Lustre: server umount lustre-MDT0001 complete [ 1580.120951] Lustre: server umount lustre-OST0000 complete [ 1582.971521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1583.120276] LustreError: 100322:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1583.129613] LustreError: 100322:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1583.145224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1584.479352] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1587.404906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1587.537242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1588.960266] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1590.602889] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1591.697753] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1593.733900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1595.504393] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1597.161182] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1608.166320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 1617.375748] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1617.378114] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1617.381954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1618.403303] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1620.426491] Lustre: server umount lustre-OST0000 complete [ 1623.520640] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1623.521106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1623.529699] Lustre: Skipped 2 previous similar messages [ 1627.799509] Lustre: server umount lustre-MDT0000 complete [ 1628.640829] LustreError: 100322:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1628.645181] LustreError: 100322:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1629.056081] LustreError: 100304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778840580 with bad export cookie 2033405281832939080 [ 1629.058047] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1629.062731] LustreError: 100304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1629.176791] Lustre: server umount lustre-MDT0001 complete [ 1632.815530] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1633.893749] Key type lgssc unregistered [ 1634.028329] LNet: 103514:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1634.030657] LNetError: 103514:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1634.037291] LNet: Removed LNI 192.168.204.126@tcp [ 1634.349938] Key type .llcrypt unregistered [ 1634.351073] Key type ._llcrypt unregistered [ 1643.855360] Key type ._llcrypt registered [ 1643.856355] Key type .llcrypt registered [ 1643.894931] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1644.312586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1644.324697] alg: No test for adler32 (adler32-zlib) [ 1645.184501] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 1645.271501] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1646.855093] Key type lgssc registered [ 1647.200944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1649.868588] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1649.875270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1651.024158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1652.174994] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1654.769356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1654.870554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1656.075395] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1657.793326] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1658.849679] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1660.856225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1660.898293] LustreError: 106484:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1660.904199] LustreError: 106484:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1660.951362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1662.667483] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1664.439338] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1665.064330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 1672.623559] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1673.434750] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1675.233373] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1675.234637] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1675.236401] LustreError: Skipped 1 previous similar message [ 1675.240869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1675.245275] Lustre: Skipped 1 previous similar message [ 1680.354094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1680.356152] Lustre: Skipped 1 previous similar message [ 1680.525608] Lustre: server umount lustre-OST0000 complete [ 1685.473027] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1685.473094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1685.473783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1685.477127] Lustre: Skipped 2 previous similar messages [ 1690.592211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1690.594290] Lustre: Skipped 2 previous similar messages [ 1695.712070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1695.714316] Lustre: Skipped 1 previous similar message [ 1696.223084] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1696.277159] Lustre: server umount lustre-MDT0000 complete [ 1697.476244] LustreError: 104870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778840648 with bad export cookie 7032364801921689204 [ 1697.479255] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1697.482793] LustreError: 104870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1697.599194] Lustre: server umount lustre-MDT0001 complete [ 1701.300314] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1702.349287] Key type lgssc unregistered [ 1702.483389] LNet: 108170:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1702.486095] LNetError: 108170:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1702.497399] LNet: Removed LNI 192.168.204.126@tcp [ 1702.832466] Key type .llcrypt unregistered [ 1702.834070] Key type ._llcrypt unregistered [ 1709.503057] Key type ._llcrypt registered [ 1709.504342] Key type .llcrypt registered [ 1709.540432] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1709.892684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1709.911753] alg: No test for adler32 (adler32-zlib) [ 1710.753825] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 1710.830456] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1712.407154] Key type lgssc registered [ 1712.747326] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1715.244740] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1715.251233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1716.393169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1717.647598] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1720.304488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1720.416082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1721.661613] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1723.376439] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1724.499133] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1726.547051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1726.659257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1727.718673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 1728.512289] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1730.287982] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1735.138215] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1735.139054] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1735.139150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1735.141910] Lustre: Skipped 1 previous similar message [ 1739.919046] Lustre: server umount lustre-OST0000 complete [ 1745.376491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1745.377099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1745.380569] Lustre: Skipped 1 previous similar message [ 1745.383280] Lustre: Skipped 1 previous similar message [ 1747.289489] Lustre: server umount lustre-MDT0000 complete [ 1748.487127] LustreError: 109333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778840699 with bad export cookie 2464310242177414031 [ 1748.490137] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1748.490302] LustreError: 109333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1748.602554] Lustre: server umount lustre-MDT0001 complete [ 1752.032430] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1753.011718] Key type lgssc unregistered [ 1753.130359] LNet: 112531:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1753.133300] LNetError: 112531:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1753.141270] LNet: Removed LNI 192.168.204.126@tcp [ 1753.423359] Key type .llcrypt unregistered [ 1753.424407] Key type ._llcrypt unregistered [ 1761.786839] Key type ._llcrypt registered [ 1761.788274] Key type .llcrypt registered [ 1761.825212] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1762.134474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1762.175893] alg: No test for adler32 (adler32-zlib) [ 1763.014725] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 1763.087185] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1764.663149] Key type lgssc registered [ 1764.996116] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1767.390577] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1767.395760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1768.538724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1769.752217] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1772.455217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1772.562268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1773.789315] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1775.442509] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1776.496677] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1778.672377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1778.779474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1780.547628] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1782.150884] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1794.150632] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 1800.282258] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1801.558064] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1801.560905] Lustre: Skipped 2 previous similar messages [ 1802.894851] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1802.897261] Lustre: Skipped 2 previous similar messages [ 1806.843077] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1806.856197] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1809.159882] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.26@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1809.163341] Lustre: Skipped 2 previous similar messages [ 1811.964327] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.26@tcp 192.168.0.[2,10]@tcp [ 1813.247024] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.26/24@tcp in log lustre-MDT0000 [ 1813.250671] Lustre: Skipped 2 previous similar messages [ 1820.153838] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.26/24@tcp [ 1820.156893] Lustre: Skipped 1 previous similar message [ 1823.199973] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1823.203515] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1823.208088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1825.252260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1825.252314] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1825.258976] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1828.623757] Lustre: server umount lustre-OST0000 complete [ 1830.368663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1830.369466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1830.374318] Lustre: Skipped 1 previous similar message [ 1830.376953] Lustre: Skipped 1 previous similar message [ 1835.487946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1835.490205] Lustre: Skipped 2 previous similar messages [ 1840.608092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1840.610267] Lustre: Skipped 1 previous similar message [ 1844.191153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1844.259108] Lustre: server umount lustre-MDT0000 complete [ 1845.518331] LustreError: 113889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778840796 with bad export cookie 10584702779883151015 [ 1845.521104] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1845.523024] LustreError: 113889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1845.653112] Lustre: server umount lustre-MDT0001 complete [ 1849.315967] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1850.392174] Key type lgssc unregistered [ 1850.524311] LNet: 118647:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1850.526646] LNetError: 118647:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1850.539341] LNet: Removed LNI 192.168.204.126@tcp [ 1850.827744] Key type .llcrypt unregistered [ 1850.828857] Key type ._llcrypt unregistered [ 1861.164122] Key type ._llcrypt registered [ 1861.165198] Key type .llcrypt registered [ 1861.205122] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1861.590313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1861.635877] alg: No test for adler32 (adler32-zlib) [ 1862.498751] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 1862.589198] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1864.175154] Key type lgssc registered [ 1864.585639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1867.317473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1867.324135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1868.480693] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1868.485346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1868.488722] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.26/24@tcp [ 1869.860884] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1872.839019] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1872.953183] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1872.963092] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.26/24@tcp [ 1872.979638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1874.292510] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1876.138414] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1877.348946] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1879.561550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1879.677033] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1881.551221] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1883.298521] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1894.055878] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 1903.584622] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1903.587150] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1903.591313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1904.608693] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1904.610792] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1904.610893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1907.725909] Lustre: server umount lustre-OST0000 complete [ 1909.728801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1909.729544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1909.733133] Lustre: Skipped 1 previous similar message [ 1909.739203] Lustre: Skipped 2 previous similar messages [ 1914.848248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1914.852787] Lustre: Skipped 1 previous similar message [ 1915.097401] Lustre: server umount lustre-MDT0000 complete [ 1916.344132] LustreError: 120151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778840867 with bad export cookie 17120358802961877192 [ 1916.347151] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1916.347741] LustreError: 120151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1916.469140] Lustre: server umount lustre-MDT0001 complete [ 1919.977633] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1920.962315] Key type lgssc unregistered [ 1921.085362] LNet: 123417:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1921.087651] LNetError: 123417:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1921.097273] LNet: Removed LNI 192.168.204.126@tcp [ 1921.426052] Key type .llcrypt unregistered [ 1921.427092] Key type ._llcrypt unregistered [ 1929.838329] Key type ._llcrypt registered [ 1929.839987] Key type .llcrypt registered [ 1929.876102] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 1935.754621] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1936.104495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1936.137749] alg: No test for adler32 (adler32-zlib) [ 1936.990043] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 1937.073076] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1938.647130] Key type lgssc registered [ 1938.994398] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1949.756102] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1949.763509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1950.847959] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1950.859434] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1950.894699] Lustre: lustre-MDT0000: new disk, initializing [ 1950.919634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1950.926573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1952.211244] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1956.530937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1956.582344] Lustre: 126755:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1956.614892] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1956.617772] Lustre: Skipped 1 previous similar message [ 1956.657116] Lustre: lustre-MDT0001: new disk, initializing [ 1956.694664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1956.712044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1956.715435] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1958.240237] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1961.998408] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1963.340237] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1965.730427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1965.861701] Lustre: lustre-OST0000: new disk, initializing [ 1965.863880] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1965.868036] Lustre: 128494:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1965.909228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1968.025182] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1971.099386] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1972.496128] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1973.240700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1973.244361] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1973.282447] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1973.603315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 1975.023979] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1975.108640] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1978.338773] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1978.339744] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1978.341770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1978.347898] Lustre: Skipped 1 previous similar message [ 1981.905805] Lustre: server umount lustre-OST0000 complete [ 1983.456976] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1983.457681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1983.461553] Lustre: Skipped 1 previous similar message [ 1983.465840] Lustre: Skipped 2 previous similar messages [ 1988.576245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1988.578611] Lustre: Skipped 1 previous similar message [ 1989.342431] Lustre: server umount lustre-MDT0000 complete [ 1990.619561] LustreError: 126748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778840942 with bad export cookie 17367823225826852021 [ 1990.622997] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1990.624902] LustreError: 126748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1990.750262] Lustre: server umount lustre-MDT0001 complete [ 1993.667058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1993.836711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1995.128451] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1997.957336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1998.073575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1999.347463] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2001.162286] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2002.343678] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2004.592232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2006.467975] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2008.176052] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2009.495467] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2019.748823] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 2020.978921] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2021.050016] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2024.321659] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2024.396648] Lustre: lustre-OST0001: new disk, initializing [ 2024.400386] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2024.405420] Lustre: 133633:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2024.450312] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2024.453453] Lustre: Skipped 1 previous similar message [ 2026.261808] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2029.491323] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2032.638178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2032.641468] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2032.665457] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2033.654822] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 2035.068562] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2035.144351] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2036.319655] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90c410bc8000.ost_server_uuid 50 [ 2036.822619] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90c410bc8000.ost_server_uuid in FULL state after 0 sec [ 2042.850833] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2042.852759] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2042.854642] LustreError: Skipped 1 previous similar message [ 2042.858731] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2042.862809] Lustre: Skipped 1 previous similar message [ 2044.732178] Lustre: server umount lustre-OST0001 complete [ 2047.969624] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2047.971370] LustreError: 132157:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2047.971738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2047.973094] Lustre: Skipped 2 previous similar messages [ 2047.980177] LustreError: 132157:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 2053.090543] LustreError: 132157:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2053.098063] LustreError: 132157:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2058.209399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2058.211236] LustreError: 132665:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2058.211537] Lustre: Skipped 4 previous similar messages [ 2058.216052] LustreError: 132665:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2060.255100] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2060.304914] Lustre: server umount lustre-OST0000 complete [ 2063.328952] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2067.741352] Lustre: server umount lustre-MDT0000 complete [ 2068.448794] LustreError: 130567:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2068.454772] LustreError: 130567:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2068.981181] LustreError: 133632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778841020 with bad export cookie 17367823225826852875 [ 2068.984284] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2068.986905] LustreError: 133632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2069.102702] Lustre: server umount lustre-MDT0001 complete [ 2075.786687] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2076.899392] Key type lgssc unregistered [ 2077.034456] LNet: 136499:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2077.037191] LNetError: 136499:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2077.045424] LNet: Removed LNI 192.168.204.126@tcp [ 2077.361173] Key type .llcrypt unregistered [ 2077.362728] Key type ._llcrypt unregistered [ 2090.028148] Key type ._llcrypt registered [ 2090.029220] Key type .llcrypt registered [ 2090.070824] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 2095.957091] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2096.254749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2096.266535] alg: No test for adler32 (adler32-zlib) [ 2097.121354] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 2097.209978] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2098.791167] Key type lgssc registered [ 2099.164302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2110.268588] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2110.276507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2111.364400] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2111.375220] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2111.409494] Lustre: lustre-MDT0000: new disk, initializing [ 2111.449740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2111.457568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2112.742949] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2116.854607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2116.892174] Lustre: 140588:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2116.913582] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2116.916373] Lustre: Skipped 1 previous similar message [ 2116.957916] Lustre: lustre-MDT0001: new disk, initializing [ 2116.990411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2117.004747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2117.008044] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2118.302762] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2121.353045] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2122.605728] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2124.856443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2124.960237] Lustre: lustre-OST0000: new disk, initializing [ 2124.961886] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2124.964916] Lustre: 142327:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2124.992606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2126.648526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2126.653576] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2126.687449] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2126.862607] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2129.904874] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2131.850778] Lustre: Failing over lustre-OST0000 [ 2131.918831] Lustre: server umount lustre-OST0000 complete [ 2144.867711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2144.989706] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2144.997478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2145.006867] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2146.518099] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2146.936533] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2147.808161] Lustre: 138816:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778841083/real 1778841083] req@0000000020e613d6 x1865250030772608/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1778841099 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2148.242463] Lustre: Failing over lustre-MDT0000 [ 2148.351566] Lustre: server umount lustre-MDT0000 complete [ 2149.279107] Lustre: 138817:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778841083/real 1778841083] req@0000000028837afb x1865250030773248/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1778841099 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2149.290986] Lustre: 138817:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2150.369546] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2150.375964] Lustre: Skipped 3 previous similar messages [ 2153.439131] Lustre: 138816:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778841088/real 1778841088] req@000000000a3fafbd x1865250030774656/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1778841104 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2155.488052] LustreError: 140601:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2155.493342] LustreError: 140601:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 7 previous similar messages [ 2156.747464] LustreError: 140597:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2158.559130] Lustre: 138816:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778841093/real 1778841093] req@000000004c791620 x1865250030775808/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1778841109 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2160.608506] LustreError: 140600:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2160.613638] LustreError: 140600:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2161.412695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2161.450424] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2161.454488] Lustre: 140592:0:(mgc_request.c:1897:mgc_process_log()) MGC192.168.204.126@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2161.532142] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2161.548982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2161.571499] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2161.867187] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2162.841042] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2166.757137] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2166.763735] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2166.766939] Lustre: Skipped 1 previous similar message [ 2166.784982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2171.874603] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2171.876282] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2171.876808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2171.879868] Lustre: Skipped 1 previous similar message [ 2173.903101] Lustre: server umount lustre-OST0000 complete [ 2176.992917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2176.992938] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2176.999640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2177.001318] LustreError: Skipped 1 previous similar message [ 2177.004932] Lustre: Skipped 2 previous similar messages [ 2181.268174] Lustre: server umount lustre-MDT0000 complete [ 2182.111948] LustreError: 140600:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2182.116539] LustreError: 140600:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2182.462197] LustreError: 145077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778841133 with bad export cookie 10486880372208414681 [ 2182.465803] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2182.466765] LustreError: 145077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2182.581496] Lustre: server umount lustre-MDT0001 complete [ 2185.948302] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2186.988485] Key type lgssc unregistered [ 2187.115400] LNet: 145811:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2187.119325] LNetError: 145811:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2187.133356] LNet: Removed LNI 192.168.204.126@tcp [ 2187.426791] Key type .llcrypt unregistered [ 2187.427851] Key type ._llcrypt unregistered [ 2207.139765] Key type ._llcrypt registered [ 2207.140926] Key type .llcrypt registered [ 2207.180427] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2207.644176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2207.660645] alg: No test for adler32 (adler32-zlib) [ 2208.525125] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 2208.614707] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2210.199183] Key type lgssc registered [ 2210.545497] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2213.078117] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2213.085346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2214.242285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2215.533982] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2218.525318] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2220.477120] LustreError: 147341:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2220.480296] LustreError: 147341:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2220.482840] LustreError: 147341:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 2220.591694] Lustre: server umount lustre-MDT0000 complete [ 2225.334378] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 2228.329886] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2239.763605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2239.857600] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2239.870350] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2239.907729] Lustre: lustre-MDT0000: new disk, initializing [ 2239.933155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2239.939706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2241.269623] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2245.444022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2245.486117] Lustre: 150489:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2245.509122] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2245.511387] Lustre: Skipped 1 previous similar message [ 2245.548723] Lustre: lustre-MDT0001: new disk, initializing [ 2245.583648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2245.595366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2245.598500] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2246.917862] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2249.966637] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2251.114411] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2253.303341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2253.404443] Lustre: lustre-OST0000: new disk, initializing [ 2253.406572] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2253.410032] Lustre: 152233:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2253.450488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2255.262657] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2258.234027] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2260.470533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2260.473788] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2260.504442] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2265.433520] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2265.982816] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 06:33:36 (1778841216) [ 2332.127641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2332.127861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2332.132041] Lustre: Skipped 2 previous similar messages [ 2332.134157] Lustre: Skipped 1 previous similar message [ 2337.247513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2337.249625] Lustre: Skipped 3 previous similar messages [ 2337.887910] Lustre: server umount lustre-MDT0000 complete [ 2339.053478] LustreError: 151269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778841290 with bad export cookie 17052632763582437007 [ 2339.054653] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2339.057623] LustreError: 151269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2342.367373] LustreError: 150497:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2342.367474] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2342.371952] LustreError: 150497:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2342.379336] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2344.415362] LustreError: 150498:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2344.415374] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2345.244974] Lustre: server umount lustre-MDT0001 complete [ 2352.594533] Lustre: server umount lustre-OST0000 complete [ 2355.402874] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 2358.196081] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2369.569357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2369.655076] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2369.666305] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2369.701345] Lustre: lustre-MDT0000: new disk, initializing [ 2369.726709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2369.732727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2370.963262] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2374.979745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2375.019661] Lustre: 156713:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2375.041961] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2375.044391] Lustre: Skipped 1 previous similar message [ 2375.082332] Lustre: lustre-MDT0001: new disk, initializing [ 2375.123239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2375.126898] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2376.447765] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2379.468922] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2380.633928] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2382.894318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2383.017313] Lustre: 158453:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2383.051510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2383.054274] Lustre: Skipped 1 previous similar message [ 2384.631280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2384.634695] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2384.667241] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2384.906829] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2388.021446] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2389.456700] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2389.547740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2390.900411] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2390.988288] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2395.104752] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2395.107309] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2395.107666] LustreError: Skipped 1 previous similar message [ 2395.112809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2395.115940] Lustre: Skipped 1 previous similar message [ 2397.717698] Lustre: server umount lustre-OST0000 complete [ 2400.224857] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2400.228812] Lustre: Skipped 2 previous similar messages [ 2405.208394] Lustre: server umount lustre-MDT0000 complete [ 2405.344116] LustreError: 156722:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2405.349355] LustreError: 156722:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2406.494997] LustreError: 156704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778841357 with bad export cookie 17052632763582595823 [ 2406.497875] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2406.498687] LustreError: 156704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2406.612576] Lustre: server umount lustre-MDT0001 complete [ 2409.291652] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 06:35:59 (1778841359) [ 2414.363845] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 2417.450996] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2428.690089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2428.779528] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2428.792845] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2428.794738] Lustre: Skipped 1 previous similar message [ 2428.829752] Lustre: lustre-MDT0000: new disk, initializing [ 2428.831386] Lustre: Skipped 1 previous similar message [ 2428.855522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2428.862553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2430.213380] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2434.434281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2434.486213] Lustre: 163020:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2434.614707] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2436.075419] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2439.239803] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2440.520944] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2443.115500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2443.246192] Lustre: 164772:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2445.129349] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2448.044817] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2452.988123] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2453.039172] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2458.080972] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2458.082137] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2458.083202] LustreError: Skipped 1 previous similar message [ 2458.083966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2458.086740] Lustre: Skipped 1 previous similar message [ 2458.088095] Lustre: Skipped 2 previous similar messages [ 2461.206958] Lustre: server umount lustre-OST0000 complete [ 2468.569994] Lustre: server umount lustre-MDT0000 complete [ 2469.771100] LustreError: 163012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778841421 with bad export cookie 17052632763582596691 [ 2469.773452] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2469.775852] LustreError: 163012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2469.877921] Lustre: server umount lustre-MDT0001 complete [ 2472.506193] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 06:37:02 (1778841422) [ 2477.382286] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 2480.334453] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2491.620371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2491.716866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2491.727711] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2491.729613] Lustre: Skipped 3 previous similar messages [ 2491.768714] Lustre: lustre-MDT0000: new disk, initializing [ 2491.770077] Lustre: Skipped 2 previous similar messages [ 2491.794792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2491.797047] Lustre: Skipped 2 previous similar messages [ 2491.804028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2491.807905] Lustre: Skipped 2 previous similar messages [ 2493.086350] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2497.162796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2497.206773] Lustre: 168937:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2497.311772] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2498.576747] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2501.627194] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2502.794459] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2504.958351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2505.061533] Lustre: 170688:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2506.908642] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2509.848526] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2509.885649] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2514.913256] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2514.916157] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2514.917457] LustreError: Skipped 1 previous similar message [ 2514.924060] Lustre: Skipped 2 previous similar messages [ 2514.927122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2514.931022] Lustre: Skipped 4 previous similar messages [ 2517.786677] Lustre: server umount lustre-OST0000 complete [ 2533.343105] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2533.410237] Lustre: server umount lustre-MDT0000 complete [ 2534.613194] LustreError: 169720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778841486 with bad export cookie 17052632763582597762 [ 2534.615705] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2534.616663] LustreError: 169720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2534.739778] Lustre: server umount lustre-MDT0001 complete [ 2538.020014] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2538.974652] Key type lgssc unregistered [ 2539.092449] LNet: 172413:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2539.095301] LNetError: 172413:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2539.109426] LNet: Removed LNI 192.168.204.126@tcp [ 2539.390840] Key type .llcrypt unregistered [ 2539.392029] Key type ._llcrypt unregistered [ 2548.076655] Key type ._llcrypt registered [ 2548.078170] Key type .llcrypt registered [ 2548.114614] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2548.470975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2548.500086] alg: No test for adler32 (adler32-zlib) [ 2549.352902] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 2549.438805] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2551.015259] Key type lgssc registered [ 2551.414431] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2553.869276] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2553.876178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2555.031895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2556.281492] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2559.011609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2559.134672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2560.415105] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2562.117179] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2563.294580] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2565.612239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2565.725726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2567.533777] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2569.262838] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2581.094879] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2596.833750] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2596.834556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2596.836152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2596.848357] Lustre: Skipped 1 previous similar message [ 2601.952740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2601.955506] Lustre: Skipped 2 previous similar messages [ 2602.385384] Lustre: server umount lustre-OST0000 complete [ 2605.535880] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2605.538321] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2605.542587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2607.072951] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2609.687389] Lustre: server umount lustre-MDT0000 complete [ 2610.887205] LustreError: 175953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778841562 with bad export cookie 363194024723530921 [ 2610.890268] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2610.890541] LustreError: 175953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2611.003457] Lustre: server umount lustre-MDT0001 complete [ 2614.429707] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2615.379148] Key type lgssc unregistered [ 2615.499384] LNet: 176987:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2615.501856] LNetError: 176987:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2615.508286] LNet: Removed LNI 192.168.204.126@tcp [ 2615.802239] Key type .llcrypt unregistered [ 2615.803312] Key type ._llcrypt unregistered [ 2624.351602] Key type ._llcrypt registered [ 2624.353436] Key type .llcrypt registered [ 2624.389434] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2624.747587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2624.778506] alg: No test for adler32 (adler32-zlib) [ 2625.623979] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 2625.703279] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2627.279175] Key type lgssc registered [ 2627.696387] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2630.333347] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2630.339404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2631.489732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2632.705424] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2635.689847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2635.809392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2637.060680] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2638.824695] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2640.016986] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2642.403070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2642.528477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2644.372028] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2646.061203] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2656.866712] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2666.464466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2666.467024] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2666.471102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2667.490490] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2667.490895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2667.495376] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2667.496888] Lustre: Skipped 2 previous similar messages [ 2669.838374] Lustre: server umount lustre-OST0000 complete [ 2671.641218] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff90c408265800.ost_server_uuid 50 [ 2672.121653] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff90c408265800.ost_server_uuid in DISCONN state after 0 sec [ 2672.647673] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2682.848197] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2682.848716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2682.851758] Lustre: Skipped 1 previous similar message [ 2682.854751] Lustre: Skipped 1 previous similar message [ 2687.968080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2687.970235] Lustre: Skipped 1 previous similar message [ 2688.720759] Lustre: server umount lustre-MDT0000 complete [ 2689.966303] LustreError: 178343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778841641 with bad export cookie 17872501401415355230 [ 2689.969161] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2689.969845] LustreError: 178343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2690.087263] Lustre: server umount lustre-MDT0001 complete [ 2692.923055] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 06:40:43 (1778841643) [ 2695.094589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2695.239711] LustreError: 181891:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2695.245863] LustreError: 181891:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2695.262376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2696.495171] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2699.241995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2699.365379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2700.640765] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2702.402728] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2703.581597] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2705.806606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2707.718473] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2709.033174] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2709.457103] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2711.742268] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2711.821372] Lustre: lustre-OST0001: new disk, initializing [ 2711.822878] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2711.825121] Lustre: 184391:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2711.855693] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2711.857694] Lustre: Skipped 1 previous similar message [ 2712.702820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2712.706323] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2712.746345] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2713.711839] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2716.745859] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2718.177888] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2718.180259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2718.181628] Lustre: Skipped 1 previous similar message [ 2723.769837] Lustre: server umount lustre-OST0000 complete [ 2725.678120] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2725.751906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2727.018152] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2727.086535] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2727.117320] LustreError: 184538:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2728.417618] LustreError: 183484:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2728.422522] LustreError: 183484:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2732.236529] LustreError: 183482:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2738.657994] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2738.658526] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2738.658769] LustreError: 183992:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2738.658789] LustreError: 183992:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 2738.660148] LustreError: Skipped 1 previous similar message [ 2738.663994] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2738.672577] Lustre: Skipped 5 previous similar messages [ 2743.185092] Lustre: server umount lustre-OST0001 complete [ 2745.311744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2748.896600] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2748.899794] Lustre: Skipped 2 previous similar messages [ 2750.545287] Lustre: server umount lustre-MDT0000 complete [ 2751.725418] LustreError: 183507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778841703 with bad export cookie 17872501401415356287 [ 2751.728493] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2751.729261] LustreError: 183507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2751.846590] Lustre: server umount lustre-MDT0001 complete [ 2759.448870] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 06:41:49 (1778841709) [ 2761.686021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2761.775139] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2761.786718] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2761.851097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2763.089387] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2765.915997] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2765.946524] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2765.959211] Lustre: 187659:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2767.258621] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2769.168588] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2770.351864] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2772.633017] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2772.701077] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2774.411271] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 2774.575071] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2776.276982] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2778.440109] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2778.464266] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2780.353452] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2782.032707] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2783.716065] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2783.723468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2783.725406] Lustre: Skipped 4 previous similar messages [ 2783.759798] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 2788.921991] Lustre: server umount lustre-OST0000 complete [ 2792.652432] LustreError: 189280:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2804.194910] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2804.198988] LustreError: Skipped 1 previous similar message [ 2805.520098] Lustre: server umount lustre-OST0001 complete [ 2819.552427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2819.556130] Lustre: Skipped 11 previous similar messages [ 2821.087180] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2821.173546] Lustre: server umount lustre-MDT0000 complete [ 2822.449192] LustreError: 187651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778841773 with bad export cookie 17872501401415357645 [ 2822.452350] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2822.453503] LustreError: 187651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2822.573969] Lustre: server umount lustre-MDT0001 complete [ 2831.628013] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 06:43:02 (1778841782) [ 2836.800147] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 2839.694472] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2851.107963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2851.212334] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2851.224592] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2851.262372] Lustre: lustre-MDT0000: new disk, initializing [ 2851.286772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2851.288636] Lustre: Skipped 3 previous similar messages [ 2851.294054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2852.670170] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2857.523252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2857.562267] Lustre: 195230:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2857.586915] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2857.589317] Lustre: Skipped 1 previous similar message [ 2857.626232] Lustre: lustre-MDT0001: new disk, initializing [ 2857.669905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2857.673517] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2858.994739] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2862.522456] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2863.696751] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2865.844241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2865.937763] Lustre: lustre-OST0000: new disk, initializing [ 2865.939950] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2865.942718] Lustre: 196974:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2867.125798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2867.129254] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2867.166965] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2867.982606] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2871.458795] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2873.050259] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2873.120322] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2874.325711] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2874.399893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2877.411813] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2877.411924] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2877.416857] LustreError: Skipped 2 previous similar messages [ 2877.425565] Lustre: Skipped 6 previous similar messages [ 2881.259971] Lustre: server umount lustre-OST0000 complete [ 2883.553034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2883.555482] Lustre: Skipped 2 previous similar messages [ 2897.375135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2897.459765] Lustre: server umount lustre-MDT0000 complete [ 2898.819898] LustreError: 195222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778841850 with bad export cookie 17872501401415359388 [ 2898.823906] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2898.827077] LustreError: 195222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2898.952033] Lustre: server umount lustre-MDT0001 complete [ 2901.985286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2902.167622] LustreError: 199040:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2902.176277] LustreError: 199040:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 7 previous similar messages [ 2903.408541] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2906.160804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2907.563603] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2909.459855] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2910.613294] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2913.322474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2915.365904] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2917.264029] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2918.932848] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2928.215722] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 2929.472448] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2929.541426] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2932.193933] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2932.198348] LustreError: Skipped 1 previous similar message [ 2936.349305] Lustre: server umount lustre-OST0000 complete [ 2938.547448] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2940.142247] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2964.437828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2964.606330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2964.610165] Lustre: Skipped 5 previous similar messages [ 2966.653772] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2968.724314] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2969.478882] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2972.761182] Lustre: server umount lustre-OST0000 complete [ 2974.296577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2974.299924] Lustre: Skipped 5 previous similar messages [ 2976.402052] Lustre: server umount lustre-MDT0000 complete [ 2977.660966] LustreError: 199021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778841929 with bad export cookie 17872501401415360256 [ 2977.663948] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2977.665392] LustreError: 199021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2977.780201] Lustre: server umount lustre-MDT0001 complete [ 2980.624677] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 06:45:31 (1778841931) [ 2982.846163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2983.008727] LustreError: 204535:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2984.498580] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2987.169551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2988.577889] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2990.343142] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2991.541144] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2993.825861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2996.034818] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2997.890148] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2999.391690] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3009.665753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 3010.943309] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3011.015494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3013.207669] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3013.275503] Lustre: lustre-OST0001: new disk, initializing [ 3013.277253] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3013.279509] Lustre: 207599:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3015.139787] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3018.118461] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3019.434235] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3020.285047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3020.288258] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3020.313311] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3020.536526] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3021.761546] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3021.828898] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3025.379676] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3025.381128] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3025.382622] LustreError: Skipped 2 previous similar messages [ 3025.384354] Lustre: Skipped 11 previous similar messages [ 3028.540866] Lustre: server umount lustre-OST0001 complete [ 3030.345459] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3030.412100] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3031.629581] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3031.692888] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3047.115978] LustreError: 206129:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3047.120810] LustreError: 206129:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 11 previous similar messages [ 3055.239744] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3057.144185] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3058.888692] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3059.388455] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3066.106058] Lustre: server umount lustre-OST0001 complete [ 3084.173942] Lustre: server umount lustre-OST0000 complete [ 3091.608717] Lustre: server umount lustre-MDT0000 complete [ 3092.798171] LustreError: 204516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778842044 with bad export cookie 17872501401415361187 [ 3092.801175] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3092.803810] LustreError: 204516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3107.295060] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3107.374558] Lustre: server umount lustre-MDT0001 complete [ 3115.024779] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 06:47:45 (1778842065) [ 3117.232554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3117.337024] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3117.350905] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3117.435419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3117.438931] Lustre: Skipped 5 previous similar messages [ 3118.792738] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3121.611457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3121.646604] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3121.656985] Lustre: 212544:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3122.980104] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3124.682745] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3125.826618] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3127.968460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3128.030640] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3129.904627] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3131.609729] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3140.716296] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3140.741485] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3142.610154] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3144.310193] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3146.898087] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 3148.215381] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3148.291885] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3149.564558] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3149.636060] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3150.772422] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90c406444800.ost_server_uuid 50 [ 3151.267624] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90c406444800.ost_server_uuid in FULL state after 0 sec [ 3151.583586] Lustre: Permanently deactivating lustre-OST0001 [ 3151.588991] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3152.367654] Lustre: Permanently reactivating lustre-OST0001 [ 3156.452393] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3156.452504] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3156.455872] Lustre: Skipped 6 previous similar messages [ 3156.459332] LustreError: Skipped 3 previous similar messages [ 3159.360770] Lustre: server umount lustre-OST0001 complete [ 3166.668323] Lustre: server umount lustre-OST0000 complete [ 3173.976400] Lustre: server umount lustre-MDT0000 complete [ 3175.357934] LustreError: 213266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778842126 with bad export cookie 17872501401415362447 [ 3175.362256] LustreError: 213266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3175.363685] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3175.515483] Lustre: server umount lustre-MDT0001 complete [ 3183.384970] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 06:48:53 (1778842133) [ 3185.856825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3185.961360] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3185.971563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3185.975027] Lustre: Skipped 5 previous similar messages [ 3187.340523] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3190.157357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3190.189142] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3190.199101] Lustre: 218645:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3191.599308] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3193.611342] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3194.715672] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3196.851996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3196.940118] Lustre: Permanently deactivating lustre-OST0000 [ 3196.945077] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3198.919270] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3200.802205] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3203.027159] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3205.010889] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3205.053091] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3205.104339] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3206.734181] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3207.288689] Lustre: Permanently reactivating lustre-OST0000 [ 3213.303840] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3213.330052] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3225.471597] Lustre: server umount lustre-OST0001 complete [ 3225.572228] LustreError: 220265:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3225.576316] LustreError: 220265:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 12 previous similar messages [ 3232.790894] Lustre: server umount lustre-OST0000 complete [ 3240.215866] Lustre: server umount lustre-MDT0000 complete [ 3241.489249] LustreError: 218637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778842192 with bad export cookie 17872501401415364078 [ 3241.492020] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3241.493659] LustreError: 218637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3241.616511] Lustre: server umount lustre-MDT0001 complete [ 3245.108429] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3246.227570] Key type lgssc unregistered [ 3246.375470] LNet: 223038:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3246.378101] LNetError: 223038:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3246.384364] LNet: Removed LNI 192.168.204.126@tcp [ 3246.723377] Key type .llcrypt unregistered [ 3246.724650] Key type ._llcrypt unregistered [ 3255.333731] Key type ._llcrypt registered [ 3255.334868] Key type .llcrypt registered [ 3255.370398] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3255.800153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3255.815958] alg: No test for adler32 (adler32-zlib) [ 3256.671195] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 3256.755726] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3258.335114] Key type lgssc registered [ 3258.684106] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3261.957752] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3261.963199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3263.137270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3264.455036] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3267.194807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3267.230691] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3267.243375] Lustre: 224453:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3267.247829] Lustre: Permanently deactivating lustre-MDT0001 [ 3267.253598] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3267.362957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3268.633344] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3270.323867] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3271.467197] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3272.172535] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3272.175247] LustreError: 224495:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3272.178960] LustreError: 224495:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3272.182133] LustreError: 224495:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 1, failed: rc = -108 [ 3273.643981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3273.746687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3275.558581] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3276.835485] LustreError: 226581:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3276.838042] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3276.844426] LustreError: 226581:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 3277.318516] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3279.547195] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3279.611578] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3281.526772] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3283.356155] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3284.965420] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3288.040167] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3288.040526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3289.195711] Lustre: Permanently reactivating lustre-MDT0001 [ 3289.202696] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3289.204710] Lustre: Skipped 2 previous similar messages [ 3291.862219] Lustre: lustre-MDT0001: Client 78a4aff3-c32c-4146-b9fc-df9e6b0bfc29 (at 192.168.204.26@tcp) reconnecting [ 3293.491618] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3294.015090] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3295.673380] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3295.740583] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3297.082392] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3297.159968] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3298.037787] Lustre: Permanently deactivating lustre-MDT0001 [ 3298.045561] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3298.049371] Lustre: Skipped 2 previous similar messages [ 3301.053099] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3308.513183] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3308.514162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3308.515261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3308.521760] Lustre: Skipped 2 previous similar messages [ 3312.919274] Lustre: server umount lustre-MDT0000 complete [ 3313.633260] LustreError: 224482:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3313.637879] LustreError: 224482:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 3314.114195] LustreError: 225187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778842265 with bad export cookie 7302944887528369734 [ 3314.116863] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3314.119292] LustreError: 225187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3314.236456] Lustre: server umount lustre-MDT0001 complete [ 3321.545043] Lustre: server umount lustre-OST0000 complete [ 3325.529317] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 06:51:15 (1778842275) [ 3329.563507] Lustre: server umount lustre-OST0001 complete [ 3331.585099] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 3334.910089] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3346.870471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3346.958869] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3346.964470] Lustre: Skipped 2 previous similar messages [ 3346.974892] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3347.014296] Lustre: lustre-MDT0000: new disk, initializing [ 3347.038902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3347.045703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3348.441802] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3353.286733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3353.330185] Lustre: 232140:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3353.358439] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3353.360527] Lustre: Skipped 1 previous similar message [ 3353.402080] Lustre: lustre-MDT0001: new disk, initializing [ 3353.452234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3353.457670] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3354.869418] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3358.300883] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3359.697691] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3362.101891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3362.231411] Lustre: lustre-OST0000: new disk, initializing [ 3362.233445] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3362.236387] Lustre: 233879:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3362.269625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3362.271866] Lustre: Skipped 1 previous similar message [ 3364.394584] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3367.373557] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3370.999698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3371.003703] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3371.034704] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3373.336688] LustreError: 232145:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3377.515022] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3377.585285] Lustre: lustre-OST0001: new disk, initializing [ 3377.586833] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3377.589100] Lustre: 234884:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3379.076694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3379.079707] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3379.109877] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3379.546292] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3382.729290] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3383.259530] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3383.431139] LustreError: 232145:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3383.439479] LustreError: 234476:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3393.527091] LustreError: 234476:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3395.804329] Lustre: lustre-OST0001: Not available for connect from 192.168.204.26@tcp (stopping) [ 3395.809796] Lustre: Skipped 3 previous similar messages [ 3399.655005] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3399.655300] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3399.660285] LustreError: Skipped 1 previous similar message [ 3399.669348] Lustre: Skipped 1 previous similar message [ 3399.670415] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3399.679692] Lustre: Skipped 1 previous similar message [ 3400.703696] Lustre: server umount lustre-OST0001 complete [ 3400.908467] LustreError: 234488:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3404.772876] LustreError: 233864:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3404.779226] LustreError: 233864:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3409.893508] LustreError: 233863:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3409.905638] LustreError: 233863:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3415.012886] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3415.022191] Lustre: Skipped 1 previous similar message [ 3415.025773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3415.028925] Lustre: Skipped 1 previous similar message [ 3420.132719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3420.133578] LustreError: 234488:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3420.137261] Lustre: Skipped 1 previous similar message [ 3420.147541] LustreError: 234488:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 3427.295141] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3427.355567] Lustre: server umount lustre-OST0000 complete [ 3430.370890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3430.370908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3430.374534] Lustre: Skipped 1 previous similar message [ 3430.378408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3430.380023] Lustre: Skipped 3 previous similar messages [ 3434.843519] Lustre: server umount lustre-MDT0000 complete [ 3436.239337] LustreError: 232917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778842387 with bad export cookie 7302944887528372555 [ 3436.243245] LustreError: 232917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3436.243813] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3442.419566] Lustre: server umount lustre-MDT0001 complete [ 3445.838456] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3446.877556] Key type lgssc unregistered [ 3447.000370] LNet: 236806:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3447.004469] LNetError: 236806:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3447.018277] LNet: Removed LNI 192.168.204.126@tcp [ 3447.301474] Key type .llcrypt unregistered [ 3447.302749] Key type ._llcrypt unregistered [ 3462.773545] Key type ._llcrypt registered [ 3462.774889] Key type .llcrypt registered [ 3462.813894] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3463.289743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3463.312595] alg: No test for adler32 (adler32-zlib) [ 3464.164904] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 3464.247165] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3465.823147] Key type lgssc registered [ 3466.154979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3468.566286] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3468.571817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3469.673121] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3469.681593] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3469.750047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3470.958656] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3473.656196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3473.688294] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3473.697878] Lustre: 238973:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3473.790395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3475.098393] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3476.833370] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3478.009951] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3480.170834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3480.253251] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3480.313329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3482.067866] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3483.693797] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3488.268153] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 3529.186596] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3529.186980] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3529.187652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3529.189016] LustreError: Skipped 1 previous similar message [ 3529.194652] Lustre: Skipped 1 previous similar message [ 3533.675296] Lustre: server umount lustre-OST0000 complete [ 3535.145543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3539.034737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3539.145129] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3540.986633] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3542.732074] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3549.415349] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3549.422832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3549.423080] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3549.435311] Lustre: Skipped 1 previous similar message [ 3580.387562] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3580.389453] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3580.392873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3580.392910] Lustre: Skipped 1 previous similar message [ 3580.394308] LustreError: Skipped 1 previous similar message [ 3580.408615] Lustre: Skipped 1 previous similar message [ 3582.199934] Lustre: server umount lustre-OST0000 complete [ 3585.505678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3585.509148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3585.512997] Lustre: Skipped 1 previous similar message [ 3585.516104] Lustre: Skipped 1 previous similar message [ 3590.624264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3590.627769] Lustre: Skipped 1 previous similar message [ 3595.744350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3598.303123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3598.417902] Lustre: server umount lustre-MDT0000 complete [ 3600.692131] LustreError: 238965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778842552 with bad export cookie 17330360816583922519 [ 3600.696506] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3600.699784] LustreError: 238965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3600.856683] Lustre: server umount lustre-MDT0001 complete [ 3605.113751] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3606.203424] Key type lgssc unregistered [ 3606.341402] LNet: 243918:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3606.343944] LNetError: 243918:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3606.351306] LNet: Removed LNI 192.168.204.126@tcp [ 3606.643250] Key type .llcrypt unregistered [ 3606.644469] Key type ._llcrypt unregistered [ 3615.552259] Key type ._llcrypt registered [ 3615.553367] Key type .llcrypt registered [ 3615.599245] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3615.967441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3615.976230] alg: No test for adler32 (adler32-zlib) [ 3616.828675] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 3616.900956] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3618.479158] Key type lgssc registered [ 3618.865549] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3621.406265] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3621.413599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3622.600993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3623.975774] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3626.785679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3626.901188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3628.180874] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3630.012739] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3631.296937] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3633.651282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3633.780887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3635.726318] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3637.595267] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3648.101799] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3657.696760] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3657.699049] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3657.703297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3658.724416] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3658.724711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3658.724885] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3658.731477] Lustre: Skipped 1 previous similar message [ 3661.456203] Lustre: server umount lustre-OST0000 complete [ 3662.815848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3662.818715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3662.823360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3668.823595] Lustre: server umount lustre-MDT0000 complete [ 3668.959904] LustreError: 245298:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3668.964901] LustreError: 245298:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 3670.012822] LustreError: 245277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778842621 with bad export cookie 11062317413703885350 [ 3670.014073] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3670.016779] LustreError: 245277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3670.138468] Lustre: server umount lustre-MDT0001 complete [ 3673.679524] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3674.760655] Key type lgssc unregistered [ 3674.880407] LNet: 249263:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3674.884062] LNetError: 249263:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3674.891270] LNet: Removed LNI 192.168.204.126@tcp [ 3675.186308] Key type .llcrypt unregistered [ 3675.188134] Key type ._llcrypt unregistered [ 3682.100254] Key type ._llcrypt registered [ 3682.101245] Key type .llcrypt registered [ 3682.141134] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3682.498144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3682.563661] alg: No test for adler32 (adler32-zlib) [ 3683.442821] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 3683.524183] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3685.111213] Key type lgssc registered [ 3685.507768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3688.445587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3688.453339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3689.634656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3690.914445] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3693.699261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3693.821558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3695.172516] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3697.019659] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3698.311796] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3700.519153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3700.636310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3702.523284] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3704.230537] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3714.979057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3724.255863] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3724.258286] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3724.262391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3725.280567] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3725.281326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3725.283032] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3729.435541] Lustre: server umount lustre-OST0000 complete [ 3735.008047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3735.010228] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3735.014223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3735.016238] Lustre: Skipped 1 previous similar message [ 3736.853665] Lustre: server umount lustre-MDT0000 complete [ 3738.112256] LustreError: 250428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778842689 with bad export cookie 3500924731660361090 [ 3738.115176] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3738.117507] LustreError: 250428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3738.232979] Lustre: server umount lustre-MDT0001 complete [ 3741.754659] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3742.819703] Key type lgssc unregistered [ 3742.940448] LNet: 253783:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3742.943077] LNetError: 253783:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3742.950342] LNet: Removed LNI 192.168.204.126@tcp [ 3743.218649] Key type .llcrypt unregistered [ 3743.219690] Key type ._llcrypt unregistered [ 3752.130593] Key type ._llcrypt registered [ 3752.132380] Key type .llcrypt registered [ 3752.175239] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3752.626997] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3752.657311] alg: No test for adler32 (adler32-zlib) [ 3753.506186] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 3753.598500] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3755.183206] Key type lgssc registered [ 3755.556839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3758.184384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3758.192460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3759.356839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3760.624722] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3763.451220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3763.574808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3764.854748] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3766.631972] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3767.772334] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3769.840753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3769.944250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3771.642382] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3773.336803] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3784.296558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3796.962298] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3796.965193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3796.969379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3799.007845] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3799.012217] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3799.018613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3800.032577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3800.036355] Lustre: Skipped 1 previous similar message [ 3802.066297] Lustre: server umount lustre-OST0000 complete [ 3804.639737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3804.641768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3804.646670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3809.436184] Lustre: server umount lustre-MDT0000 complete [ 3810.271925] LustreError: 255159:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3810.277975] LustreError: 255159:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 3810.553836] LustreError: 255140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778842761 with bad export cookie 18087035356347762250 [ 3810.557354] LustreError: 255140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3810.557914] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3810.655729] Lustre: server umount lustre-MDT0001 complete [ 3813.776915] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3814.660525] Key type lgssc unregistered [ 3814.779385] LNet: 259127:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3814.781767] LNetError: 259127:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3814.789262] LNet: Removed LNI 192.168.204.126@tcp [ 3815.045493] Key type .llcrypt unregistered [ 3815.046673] Key type ._llcrypt unregistered [ 3821.658322] Key type ._llcrypt registered [ 3821.659291] Key type .llcrypt registered [ 3821.701786] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3822.053716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3822.062085] alg: No test for adler32 (adler32-zlib) [ 3822.922478] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 3823.000083] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3824.583156] Key type lgssc registered [ 3824.946894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3827.600893] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3827.606294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3828.753668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3829.978020] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3832.823583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3832.950066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3834.284167] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3836.090656] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3837.265047] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3839.530645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3839.639686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3841.502445] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3843.200610] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3853.987605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 3863.519726] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3863.521959] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3863.525378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3864.546279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3864.546845] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3864.547753] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3868.108527] Lustre: server umount lustre-OST0000 complete [ 3869.663970] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3869.669423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3869.671539] Lustre: Skipped 1 previous similar message [ 3874.271984] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3874.274659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3874.279518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3875.484234] Lustre: server umount lustre-MDT0000 complete [ 3876.735278] LustreError: 262954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778842828 with bad export cookie 10290814430876071521 [ 3876.738449] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3876.741204] LustreError: 262954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3876.885727] Lustre: server umount lustre-MDT0001 complete [ 3880.392798] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3881.371916] Key type lgssc unregistered [ 3881.495386] LNet: 263689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3881.497773] LNetError: 263689:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3881.509268] LNet: Removed LNI 192.168.204.126@tcp [ 3881.807485] Key type .llcrypt unregistered [ 3881.808587] Key type ._llcrypt unregistered [ 3885.248209] Key type ._llcrypt registered [ 3885.249336] Key type .llcrypt registered [ 3885.285324] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 3890.549778] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 3896.527517] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3896.837822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3896.867078] alg: No test for adler32 (adler32-zlib) [ 3897.714271] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 3897.790013] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3899.375162] Key type lgssc registered [ 3899.712775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3910.449197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3910.454389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3911.536182] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3911.545691] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3911.582624] Lustre: lustre-MDT0000: new disk, initializing [ 3911.606143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3911.613074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3912.799984] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3916.817105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3916.857578] Lustre: 267171:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3916.883186] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3916.885241] Lustre: Skipped 1 previous similar message [ 3916.919554] Lustre: lustre-MDT0001: new disk, initializing [ 3916.945232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3916.955763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3916.958875] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3918.088705] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3921.066950] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3922.213136] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3924.318248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3924.408706] Lustre: lustre-OST0000: new disk, initializing [ 3924.410397] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3924.412675] Lustre: 268915:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3924.436278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3926.233870] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3927.931449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3927.936206] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3927.966747] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3929.258158] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3930.515995] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3930.586656] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3931.820285] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3931.890275] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3933.154428] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3933.154502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3933.157533] LustreError: Skipped 1 previous similar message [ 3933.163502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3933.167353] Lustre: Skipped 1 previous similar message [ 3934.539702] Lustre: server umount lustre-OST0000 complete [ 3937.759826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3937.762773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3937.767976] Lustre: Skipped 1 previous similar message [ 3937.770574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3941.839842] Lustre: server umount lustre-MDT0000 complete [ 3943.055032] LustreError: 267163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778842894 with bad export cookie 973150024635894890 [ 3943.057394] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3943.059555] LustreError: 267163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3943.163990] Lustre: server umount lustre-MDT0001 complete [ 3945.687938] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 07:01:36 (1778842896) [ 3947.797953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3947.952118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3949.133438] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3951.837566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3951.960824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3953.357752] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3955.134868] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3956.377602] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3958.743911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3960.595866] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3962.339055] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3982.816143] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3982.819229] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3982.823951] Lustre: Skipped 1 previous similar message [ 3982.825901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3982.828367] Lustre: Skipped 2 previous similar messages [ 3986.764880] Lustre: server umount lustre-OST0000 complete [ 3987.935590] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3987.937904] LustreError: Skipped 1 previous similar message [ 3987.939210] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3987.943470] Lustre: Skipped 1 previous similar message [ 3987.945584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3987.947280] Lustre: Skipped 2 previous similar messages [ 3994.079888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3994.082712] Lustre: Skipped 3 previous similar messages [ 4002.271405] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4002.357538] Lustre: server umount lustre-MDT0000 complete [ 4004.320341] LustreError: 271170:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4004.329647] LustreError: 271170:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 4004.467352] LustreError: 273337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778842955 with bad export cookie 973150024635895758 [ 4004.474398] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4004.478074] LustreError: 273337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4004.651581] Lustre: server umount lustre-MDT0001 complete [ 4010.303955] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 4011.874495] Key type lgssc unregistered [ 4012.049579] LNet: 274376:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4012.056421] LNetError: 274376:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4012.068619] LNet: Removed LNI 192.168.204.126@tcp [ 4012.493944] Key type .llcrypt unregistered [ 4012.496173] Key type ._llcrypt unregistered [ 4029.175909] Key type ._llcrypt registered [ 4029.177884] Key type .llcrypt registered [ 4029.241768] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 4036.423854] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4036.815142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4036.873215] alg: No test for adler32 (adler32-zlib) [ 4037.739459] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 4037.842524] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 4039.439538] Key type lgssc registered [ 4039.921116] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4057.175847] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4057.190830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4058.281672] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4058.292959] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4058.324192] Lustre: lustre-MDT0000: new disk, initializing [ 4058.347467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4058.354702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4059.713766] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4064.244487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4064.292677] Lustre: 278138:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4064.327160] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4064.329925] Lustre: Skipped 1 previous similar message [ 4064.370140] Lustre: lustre-MDT0001: new disk, initializing [ 4064.411513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4064.427575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4064.432345] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4065.789080] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4069.114498] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4070.485972] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4073.044624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4073.166065] Lustre: lustre-OST2710: new disk, initializing [ 4073.168688] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4073.171974] Lustre: 279880:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4073.210987] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4075.346230] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4078.749958] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4081.281486] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4081.355143] Lustre: lustre-OST03e8: new disk, initializing [ 4081.357799] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4081.361648] Lustre: 280820:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4081.402617] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4081.834630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4081.844938] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4081.850918] Lustre: Skipped 1 previous similar message [ 4081.911408] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4081.925200] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4083.502041] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4085.392585] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4089.295605] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4089.378748] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4090.750880] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4090.832049] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4092.265686] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4092.367710] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4093.856184] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4093.938387] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4095.455923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4095.459359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4095.464639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4097.504920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4097.506785] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4097.512360] Lustre: Skipped 1 previous similar message [ 4101.220024] Lustre: server umount lustre-MDT0000 complete [ 4102.732336] LustreError: 278133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778843054 with bad export cookie 2207471080856687137 [ 4102.735988] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4102.740759] LustreError: 278133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4102.880370] Lustre: server umount lustre-MDT0001 complete [ 4113.935524] Lustre: server umount lustre-OST2710 complete [ 4125.535141] Lustre: server umount lustre-OST03e8 complete [ 4132.324898] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 4135.976437] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4148.428749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4148.515450] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4148.531294] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4148.573057] Lustre: lustre-MDT0000: new disk, initializing [ 4148.605101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4148.613812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4148.617746] Lustre: Skipped 1 previous similar message [ 4150.052339] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4154.553236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4154.595819] Lustre: 285854:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4154.694629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4154.698299] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4156.114041] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4159.255661] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4160.480330] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4162.926276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4163.047638] Lustre: lustre-OST0000: new disk, initializing [ 4163.049763] Lustre: Skipped 1 previous similar message [ 4163.051564] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4163.054418] Lustre: Skipped 2 previous similar messages [ 4163.057059] Lustre: 287594:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4163.091751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4163.095252] Lustre: Skipped 1 previous similar message [ 4165.300580] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4165.377911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4165.383083] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4165.439780] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4168.839872] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4170.237878] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4170.323565] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4171.663145] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4171.741224] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4174.481367] Lustre: server umount lustre-OST0000 complete [ 4175.839965] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4175.843805] Lustre: Skipped 1 previous similar message [ 4175.845696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4175.847586] Lustre: Skipped 3 previous similar messages [ 4180.447785] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4180.450638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4180.455148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4186.080155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4186.082843] Lustre: Skipped 2 previous similar messages [ 4190.175107] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4190.234308] Lustre: server umount lustre-MDT0000 complete [ 4191.200832] LustreError: 285866:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4191.206277] LustreError: 285866:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 6 previous similar messages [ 4191.403186] LustreError: 285847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778843142 with bad export cookie 2207471080856688705 [ 4191.406091] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4191.407755] LustreError: 285847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4200.976969] Lustre: server umount lustre-MDT0001 complete [ 4203.609359] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 07:05:54 (1778843154) [ 4204.079711] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4204.641131] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 07:05:55 (1778843155) [ 4205.582599] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4214.192299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4214.292979] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4214.300209] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4214.361601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4215.628488] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4218.351277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4218.382614] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4218.392149] Lustre: 291286:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4219.827037] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4235.618104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4235.685494] Lustre: Denying initial registration attempt from nid 192.168.204.126@tcp, specified as failover [ 4235.689178] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4235.694105] LustreError: 292703:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 4235.696160] LustreError: 292703:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 4235.698582] LustreError: 292703:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4235.719759] Lustre: server umount lustre-OST0000 complete [ 4235.721136] LustreError: 292703:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4239.327898] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4239.330702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4239.334859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4244.055104] Lustre: server umount lustre-MDT0000 complete [ 4245.316178] LustreError: 291276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778843196 with bad export cookie 2207471080856689587 [ 4245.318990] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4245.320679] LustreError: 291276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4245.447110] Lustre: server umount lustre-MDT0001 complete [ 4248.918338] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 4249.920644] Key type lgssc unregistered [ 4250.038305] LNet: 293807:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4250.041390] LNetError: 293807:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4250.051341] LNet: Removed LNI 192.168.204.126@tcp [ 4250.333936] Key type .llcrypt unregistered [ 4250.335058] Key type ._llcrypt unregistered [ 4253.809947] Key type ._llcrypt registered [ 4253.811050] Key type .llcrypt registered [ 4253.856447] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4254.293700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4254.363549] alg: No test for adler32 (adler32-zlib) [ 4255.222479] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 4255.301047] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 4256.887211] Key type lgssc registered [ 4257.284881] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4281.146159] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4284.191856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4284.198262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4285.302381] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4285.312934] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4285.367203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4286.809331] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4289.791764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4289.829864] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4289.844152] Lustre: 296510:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4289.947844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4291.330767] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4293.196537] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4294.551310] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4309.922561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4309.997422] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4310.066640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4312.133468] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4314.113629] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4314.916951] Lustre: server umount lustre-OST0000 complete [ 4318.189692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4318.190743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4318.193476] Lustre: Skipped 1 previous similar message [ 4322.327335] Lustre: server umount lustre-MDT0000 complete [ 4323.505038] LustreError: 296502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778843274 with bad export cookie 6096170207968344983 [ 4323.507857] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4323.508776] LustreError: 296502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4323.601519] Lustre: server umount lustre-MDT0001 complete [ 4327.003435] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 4327.938175] Key type lgssc unregistered [ 4328.069375] LNet: 299719:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4328.071922] LNetError: 299719:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4328.081241] LNet: Removed LNI 192.168.204.126@tcp [ 4328.348389] Key type .llcrypt unregistered [ 4328.349624] Key type ._llcrypt unregistered [ 4337.638395] Key type ._llcrypt registered [ 4337.640486] Key type .llcrypt registered [ 4337.684712] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4338.106827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4338.156711] alg: No test for adler32 (adler32-zlib) [ 4339.011989] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 4339.108109] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 4340.687135] Key type lgssc registered [ 4341.066614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4343.900450] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4343.908475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4345.093747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4346.503704] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4349.624620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4349.757904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4351.106714] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4352.990302] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4354.246064] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4358.692670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4358.915133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4360.992528] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4362.823781] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4378.327598] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 4380.643801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4380.645365] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4380.646250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4380.648278] Lustre: Skipped 2 previous similar messages [ 4382.122253] Lustre: server umount lustre-MDT0000 complete [ 4383.439167] LustreError: 301092:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4383.451205] LustreError: 301092:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4383.608737] LustreError: 301078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778843335 with bad export cookie 276555086056791163 [ 4383.611965] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4383.615371] LustreError: 301078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4385.761229] LustreError: 301093:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4385.763792] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4385.781037] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4385.784113] Lustre: Skipped 2 previous similar messages [ 4388.556275] LustreError: 302709:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4388.563480] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.26@tcp (stopping) [ 4389.842619] Lustre: server umount lustre-MDT0001 complete [ 4392.683306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4396.216389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4409.311648] LustreError: 300406:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000048c09248 x1865252381679488/t0(0) o250->MGC192.168.204.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4409.459966] LustreError: 302699:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4409.468052] LustreError: 302699:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4409.491661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4409.516151] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4411.040864] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4413.601054] LustreError: 304367:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4413.606365] LustreError: 304367:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4414.246416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4414.379586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4415.810091] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4417.762799] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4419.493065] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4424.468852] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 6 sec [ 4425.737157] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4429.394553] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4434.915117] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4434.917912] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4434.919291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4434.923152] Lustre: Skipped 1 previous similar message [ 4434.926030] Lustre: Skipped 1 previous similar message [ 4445.151175] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4445.156741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4445.159089] Lustre: Skipped 4 previous similar messages [ 4445.227431] Lustre: server umount lustre-OST0000 complete [ 4450.273221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4450.273425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4450.282022] Lustre: Skipped 1 previous similar message [ 4452.771181] Lustre: server umount lustre-MDT0000 complete [ 4454.293470] LustreError: 302714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778843405 with bad export cookie 276555086056798317 [ 4454.295222] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4454.298469] LustreError: 302714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4454.442142] Lustre: server umount lustre-MDT0001 complete [ 4458.523812] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 4459.673729] Key type lgssc unregistered [ 4459.808433] LNet: 306794:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4459.811657] LNetError: 306794:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4459.820467] LNet: Removed LNI 192.168.204.126@tcp [ 4460.132175] Key type .llcrypt unregistered [ 4460.133871] Key type ._llcrypt unregistered [ 4470.103808] Key type ._llcrypt registered [ 4470.104915] Key type .llcrypt registered [ 4470.145939] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4470.535917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4470.592130] alg: No test for adler32 (adler32-zlib) [ 4471.462472] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 4471.555046] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 4473.143176] Key type lgssc registered [ 4473.608321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4476.658337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4476.666620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4477.855437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4479.334105] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4482.660595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4482.827408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4484.458573] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4486.443312] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4487.750606] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4492.897521] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4492.898588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4492.903887] Lustre: Skipped 1 previous similar message [ 4494.930459] Lustre: server umount lustre-MDT0000 complete [ 4496.222338] LustreError: 308153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778843447 with bad export cookie 16436506613727380447 [ 4496.224990] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4496.226778] LustreError: 308153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4496.340943] Lustre: server umount lustre-MDT0001 complete [ 4501.728600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4501.841214] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4501.854361] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4501.935105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4503.451108] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4506.939105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4506.997045] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4507.015068] Lustre: 310529:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4507.158793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4508.846916] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4511.146768] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4512.626641] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4517.926948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4518.045388] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4518.052146] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4518.057400] LustreError: 312184:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 4518.061811] LustreError: 312184:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 4518.065352] LustreError: 312184:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4518.100633] Lustre: server umount lustre-OST0000 complete [ 4518.102618] LustreError: 312184:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4523.494444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4523.586687] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4523.673576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4526.068123] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4528.296782] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4529.183648] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 4531.607706] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4531.645017] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4531.682602] Lustre: lustre-OST0001: new disk, initializing [ 4531.685504] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4531.689815] Lustre: 313611:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4531.733816] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4534.023331] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4536.187703] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4537.464380] Lustre: server umount lustre-OST0001 complete [ 4538.398651] LustreError: 313289:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4538.410448] LustreError: 313289:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4539.360239] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4539.361710] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4539.364170] LustreError: Skipped 1 previous similar message [ 4539.372063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4539.377987] Lustre: Skipped 2 previous similar messages [ 4543.457915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4543.458733] LustreError: 312651:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4543.460371] Lustre: Skipped 1 previous similar message [ 4543.466964] LustreError: 312651:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4545.174860] Lustre: server umount lustre-OST0000 complete [ 4548.064133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4548.067922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4548.074451] Lustre: Skipped 1 previous similar message [ 4548.076844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4552.831129] Lustre: server umount lustre-MDT0000 complete [ 4553.697086] LustreError: 310540:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4553.708068] LustreError: 310540:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4554.458275] LustreError: 313610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778843505 with bad export cookie 16436506613727380986 [ 4554.460070] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4554.463133] LustreError: 313610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4554.612838] Lustre: server umount lustre-MDT0001 complete [ 4558.834937] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 4560.112996] Key type lgssc unregistered [ 4560.262214] LNet: 315434:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4560.266432] LNetError: 315434:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4560.277770] LNet: Removed LNI 192.168.204.126@tcp [ 4560.632435] Key type .llcrypt unregistered [ 4560.634257] Key type ._llcrypt unregistered [ 4594.375184] Key type ._llcrypt registered [ 4594.376647] Key type .llcrypt registered [ 4594.419668] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 4601.296326] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4601.770209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4601.780095] alg: No test for adler32 (adler32-zlib) [ 4602.652907] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 4602.749251] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 4604.335161] Key type lgssc registered [ 4604.755806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4618.433666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4618.462427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4619.565206] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4619.584685] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4619.627605] Lustre: lustre-MDT0000: new disk, initializing [ 4619.674187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4619.685032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4621.474620] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4626.544315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4626.596684] Lustre: 320303:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4626.629518] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4626.632569] Lustre: Skipped 1 previous similar message [ 4626.677748] Lustre: lustre-MDT0001: new disk, initializing [ 4626.717628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4626.733784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4626.738885] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4628.330987] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4631.954472] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4633.377221] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4636.238659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4636.364839] Lustre: lustre-OST0000: new disk, initializing [ 4636.367273] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4636.370967] Lustre: 322046:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4636.402470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4638.664482] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4642.044547] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4643.325817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4643.330543] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4643.372259] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4643.616593] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4643.706449] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4645.253554] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4645.339512] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4648.419579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4648.420537] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4648.422485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4648.425171] Lustre: Skipped 1 previous similar message [ 4652.250239] Lustre: server umount lustre-OST0000 complete [ 4657.632140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4657.636040] LustreError: Skipped 1 previous similar message [ 4657.638323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4657.644653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4657.647103] Lustre: Skipped 1 previous similar message [ 4658.656429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4658.656721] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4663.776600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4663.780558] Lustre: Skipped 2 previous similar messages [ 4667.872609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4667.875662] Lustre: Skipped 1 previous similar message [ 4668.383130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4668.467111] Lustre: server umount lustre-MDT0000 complete [ 4669.865214] LustreError: 321083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778843621 with bad export cookie 1348112262261034061 [ 4669.867414] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4669.869753] LustreError: 321083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4669.999618] Lustre: server umount lustre-MDT0001 complete [ 4673.244853] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 07:13:43 (1778843623) [ 4676.016425] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 07:13:46 (1778843626) [ 4678.999686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4679.194381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4680.694626] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4684.045288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4684.200195] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4685.665786] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4687.807520] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4689.297456] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4692.054652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4692.213459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4694.436639] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4696.534947] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4710.781391] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 4711.488644] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4712.117540] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4712.818247] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 4713.589376] Lustre: Failing over lustre-MDT0000 [ 4713.790562] Lustre: server umount lustre-MDT0000 complete [ 4714.977705] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4714.978678] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4714.982228] LustreError: 324652:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4714.987339] Lustre: Skipped 2 previous similar messages [ 4714.997984] LustreError: 324652:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 4717.272442] LustreError: 326799:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4720.096963] LustreError: 324653:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4720.104599] LustreError: 324653:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4722.379551] LustreError: 326264:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4727.351512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4727.427839] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4727.502501] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.26@tcp (not set up) [ 4727.550465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4727.586749] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4729.133577] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4732.619620] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4732.903756] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4732.917313] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4732.934372] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4734.147253] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4734.756802] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4736.763631] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 4737.543715] Lustre: Failing over lustre-MDT0000 [ 4737.749310] LustreError: 326799:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4737.755608] LustreError: 326799:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 4737.766273] Lustre: server umount lustre-MDT0000 complete [ 4738.018762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4738.018952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4738.023979] Lustre: Skipped 2 previous similar messages [ 4747.980690] LustreError: 326264:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4747.988197] LustreError: 326264:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 7 previous similar messages [ 4751.213636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4751.261443] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4751.391878] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4752.842140] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4753.101691] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4756.452848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4756.455746] Lustre: Skipped 2 previous similar messages [ 4756.464961] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4756.482489] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 4757.690296] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4758.264158] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4761.569353] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4761.570290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4761.573283] Lustre: Skipped 2 previous similar messages [ 4761.575649] Lustre: Skipped 1 previous similar message [ 4761.577124] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4766.676318] Lustre: server umount lustre-MDT0000 complete [ 4766.688438] LustreError: 324652:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4766.696612] LustreError: 324652:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 4768.069163] LustreError: 324634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778843719 with bad export cookie 1348112262261036756 [ 4768.072490] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4768.074132] LustreError: 324634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4768.209063] Lustre: server umount lustre-MDT0001 complete [ 4780.080535] Lustre: server umount lustre-OST0000 complete [ 4786.109466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4786.340484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4786.343220] Lustre: Skipped 1 previous similar message [ 4787.900900] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4791.311351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4792.908274] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4794.816044] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4796.103239] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4798.460462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4800.648775] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4802.724329] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4810.920388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 4816.357226] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4816.357462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4816.366639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4816.369024] Lustre: Skipped 1 previous similar message [ 4820.946984] Lustre: server umount lustre-OST0000 complete [ 4828.502489] Lustre: server umount lustre-MDT0000 complete [ 4829.727079] LustreError: 330414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778843781 with bad export cookie 1348112262261037505 [ 4829.730359] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4829.730665] LustreError: 330414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4829.838250] Lustre: server umount lustre-MDT0001 complete [ 4833.546478] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 4834.571811] Key type lgssc unregistered [ 4834.701366] LNet: 333628:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4834.704527] LNetError: 333628:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4834.715334] LNet: Removed LNI 192.168.204.126@tcp [ 4835.021352] Key type .llcrypt unregistered [ 4835.022986] Key type ._llcrypt unregistered [ 4845.001899] Key type ._llcrypt registered [ 4845.003656] Key type .llcrypt registered [ 4845.046737] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4845.356046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4845.364328] alg: No test for adler32 (adler32-zlib) [ 4846.231725] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 4846.320818] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 4847.903180] Key type lgssc registered [ 4848.291551] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4851.120598] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4851.129076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4852.311970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4853.810355] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4856.828549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4856.962508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4858.308980] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4860.057401] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4861.255601] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4863.530852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4863.649616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4865.605283] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4867.561582] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4876.966713] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 4887.520975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4887.523477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4887.527467] Lustre: Skipped 2 previous similar messages [ 4887.529991] Lustre: Skipped 1 previous similar message [ 4889.240737] Lustre: server umount lustre-MDT0000 complete [ 4890.647222] LustreError: 337363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778843842 with bad export cookie 16993630543698432869 [ 4890.649199] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4890.652804] LustreError: 337363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4890.777530] Lustre: server umount lustre-MDT0001 complete [ 4902.497797] Lustre: server umount lustre-OST0000 complete [ 4907.338722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4907.508225] LustreError: 338494:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4907.516556] LustreError: 338494:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4907.534281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4908.968980] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4911.988594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4912.114789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4913.499901] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4915.388221] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4916.545638] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4918.831626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4920.820526] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4922.580405] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4932.261649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4942.816748] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4942.819192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4942.820691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4942.822679] Lustre: Skipped 1 previous similar message [ 4946.322311] Lustre: server umount lustre-MDT0000 complete [ 4947.570316] LustreError: 338474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778843898 with bad export cookie 16993630543698433975 [ 4947.572766] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4947.575612] LustreError: 338474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4947.690514] Lustre: server umount lustre-MDT0001 complete [ 4959.259137] Lustre: server umount lustre-OST0000 complete [ 4964.777781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4964.943120] LustreError: 342083:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4964.967799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4964.970592] Lustre: Skipped 1 previous similar message [ 4966.369363] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4969.570945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4971.291516] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4973.437588] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4974.811330] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4977.296805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4979.322866] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4981.284894] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4989.744755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4995.041864] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4995.044655] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4995.045856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4995.050867] Lustre: Skipped 2 previous similar messages [ 4995.052856] Lustre: Skipped 2 previous similar messages [ 4998.999300] Lustre: server umount lustre-OST0000 complete [ 5000.672100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5000.675424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5000.680831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5005.280177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5005.280380] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5006.553307] Lustre: server umount lustre-MDT0000 complete [ 5007.865282] LustreError: 342064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778843959 with bad export cookie 16993630543698434983 [ 5007.868088] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5007.869984] LustreError: 342064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5007.996285] Lustre: server umount lustre-MDT0001 complete [ 5011.352875] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5012.449504] Key type lgssc unregistered [ 5012.577420] LNet: 345275:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5012.580925] LNetError: 345275:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5012.589499] LNet: Removed LNI 192.168.204.126@tcp [ 5012.909201] Key type .llcrypt unregistered [ 5012.911123] Key type ._llcrypt unregistered [ 5022.197950] Key type ._llcrypt registered [ 5022.199050] Key type .llcrypt registered [ 5022.241520] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5022.780248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5022.798252] alg: No test for adler32 (adler32-zlib) [ 5023.657230] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 5023.753978] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 5025.343202] Key type lgssc registered [ 5025.716416] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5028.252337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5028.258964] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5028.262125] LustreError: 346675:0:(osd_handler.c:8550:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5028.272202] LustreError: 346675:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5028.274588] LustreError: 346675:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5028.276845] LustreError: 346675:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5028.279283] LustreError: 346675:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5030.821648] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5030.825174] LustreError: 347072:0:(osd_handler.c:8550:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5030.830062] LustreError: 347072:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5030.833030] LustreError: 347072:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5030.835629] LustreError: 347072:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5030.838401] LustreError: 347072:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5036.939670] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5037.912581] Key type lgssc unregistered [ 5038.039389] LNet: 348055:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5038.042217] LNetError: 348055:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5038.049303] LNet: Removed LNI 192.168.204.126@tcp [ 5038.312993] Key type .llcrypt unregistered [ 5038.314785] Key type ._llcrypt unregistered [ 5044.402337] Key type ._llcrypt registered [ 5044.403698] Key type .llcrypt registered [ 5044.447448] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 5050.835441] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5051.223444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5051.259569] alg: No test for adler32 (adler32-zlib) [ 5052.129548] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 5052.221451] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 5053.815202] Key type lgssc registered [ 5054.220307] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5065.463643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5065.470661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5066.567290] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5066.578618] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5066.619890] Lustre: lustre-MDT0000: new disk, initializing [ 5066.651097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5066.659949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5068.078344] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5072.364450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5072.402565] Lustre: 351151:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5072.423724] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5072.425773] Lustre: Skipped 1 previous similar message [ 5072.458712] Lustre: lustre-MDT0001: new disk, initializing [ 5072.491453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5072.505117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5072.509842] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5073.901044] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5077.111462] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5078.434377] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5080.882367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5080.996553] Lustre: lustre-OST0000: new disk, initializing [ 5080.998737] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5081.002042] Lustre: 352892:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5081.033761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5083.079564] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5084.214069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5084.217143] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5084.242671] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5086.212771] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5087.613225] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5087.695716] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5089.051935] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5089.131292] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5094.372470] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5094.372974] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5094.378697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5094.381288] LustreError: Skipped 1 previous similar message [ 5094.385959] Lustre: Skipped 1 previous similar message [ 5099.490835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5099.493339] Lustre: Skipped 1 previous similar message [ 5104.095140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5104.155051] Lustre: server umount lustre-OST0000 complete [ 5108.704040] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5108.707149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5108.712172] Lustre: Skipped 1 previous similar message [ 5108.714236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5109.729460] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5111.637105] Lustre: server umount lustre-MDT0000 complete [ 5112.897138] LustreError: 351141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778844064 with bad export cookie 11624583279123483965 [ 5112.899844] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5112.901806] LustreError: 351141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5113.036347] Lustre: server umount lustre-MDT0001 complete [ 5115.842933] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5116.378073] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 07:21:06 (1778844066) [ 5118.579116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5118.756959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5120.034146] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5122.872234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5122.980917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5124.255363] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5125.982593] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5127.169844] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5129.441218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5131.475595] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5133.298796] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5135.546373] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5135.602852] Lustre: lustre-OST0001: new disk, initializing [ 5135.604552] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5135.606736] Lustre: 357698:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5135.635193] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5135.637355] Lustre: Skipped 1 previous similar message [ 5136.699540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5136.703635] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5136.725295] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5137.485045] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5140.596829] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5144.483821] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5144.485310] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5144.489427] Lustre: Skipped 1 previous similar message [ 5144.494759] Lustre: Skipped 3 previous similar messages [ 5148.539265] Lustre: server umount lustre-OST0001 complete [ 5149.666488] LustreError: 358393:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5149.672136] LustreError: 358393:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 5154.786832] LustreError: 358393:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5154.792480] LustreError: 358393:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5159.905630] LustreError: 356800:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5159.911139] LustreError: 356800:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5163.487896] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5163.490244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5163.494366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5163.497225] Lustre: Skipped 1 previous similar message [ 5165.027448] LustreError: 358393:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5165.032320] LustreError: 358393:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5166.542326] Lustre: server umount lustre-OST0000 complete [ 5169.119766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5169.122636] LustreError: Skipped 1 previous similar message [ 5173.982876] Lustre: server umount lustre-MDT0000 complete [ 5175.149078] LustreError: 357697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778844126 with bad export cookie 11624583279123484847 [ 5175.151554] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5175.153057] LustreError: 357697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5181.299428] Lustre: server umount lustre-MDT0001 complete [ 5185.088371] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5186.229154] Key type lgssc unregistered [ 5186.350321] LNet: 359552:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5186.352900] LNetError: 359552:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5186.363280] LNet: Removed LNI 192.168.204.126@tcp [ 5186.640821] Key type .llcrypt unregistered [ 5186.641941] Key type ._llcrypt unregistered [ 5203.409889] Key type ._llcrypt registered [ 5203.411659] Key type .llcrypt registered [ 5203.452436] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5203.930972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5203.942372] alg: No test for adler32 (adler32-zlib) [ 5204.818572] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 5204.913753] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 5206.495225] Key type lgssc registered [ 5206.883974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5209.439744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5209.445767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5210.538885] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5210.546336] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5210.611047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5211.819885] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5214.699474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5214.737520] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5214.749369] Lustre: 362308:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5214.845365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5216.159661] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5217.825461] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5219.027449] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5220.319855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5220.323298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5220.329151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5222.368130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5222.368409] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5225.807621] Lustre: server umount lustre-MDT0000 complete [ 5227.004046] LustreError: 362300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778844178 with bad export cookie 8236595677230400558 [ 5227.006396] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5227.008554] LustreError: 362300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5227.123347] Lustre: server umount lustre-MDT0001 complete [ 5230.008614] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 07:23:00 (1778844180) [ 5232.186420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5232.356963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5233.585839] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5236.226786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5236.328056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5237.612079] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5239.499987] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5240.719779] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5243.129255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5243.212731] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5243.275553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5245.311181] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5247.131365] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5250.851126] Lustre: Permanently deactivating lustre-OST0000 [ 5250.856709] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5251.951368] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5252.789214] LustreError: 367091:0:(mgs_llog.c:2031:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5252.793888] LustreError: 367091:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 5252.797354] LustreError: 367091:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5253.451861] LustreError: 367139:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 5253.455094] LustreError: 367139:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5260.182448] Lustre: server umount lustre-OST0000 complete [ 5261.082909] LustreError: 367389:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 5261.085660] LustreError: 367389:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5261.791749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5261.794259] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5261.797707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5261.799447] Lustre: Skipped 1 previous similar message [ 5266.400189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5266.400556] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5271.520391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5271.522013] Lustre: Skipped 2 previous similar messages [ 5276.127110] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5276.195245] Lustre: server umount lustre-MDT0000 complete [ 5276.640147] LustreError: 365299:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5276.647635] LustreError: 365299:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 5277.514397] LustreError: 365304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778844228 with bad export cookie 8236595677230401132 [ 5277.518308] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5277.519986] LustreError: 365304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5286.928114] Lustre: server umount lustre-MDT0001 complete [ 5289.935517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5291.227626] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5292.075662] LustreError: 368530:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5294.003893] Lustre: 368868:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.126@tcp [ 5294.673489] Lustre: server umount MGS complete [ 5298.261561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5298.410454] LustreError: 369591:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5298.414691] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5298.417346] Lustre: Skipped 1 previous similar message [ 5298.433290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5299.613392] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5302.251260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5302.344905] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5303.525946] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5305.350199] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5306.590736] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5307.221891] Lustre: Permanently reactivating lustre-OST0000 [ 5307.227769] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5307.230265] Lustre: Skipped 2 previous similar messages [ 5329.887163] LustreError: 371048:0:(osp_dev.c:695:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5333.099950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5333.217201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5333.220300] Lustre: Skipped 1 previous similar message [ 5335.088910] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5336.980261] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5349.855968] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5349.860258] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5349.864353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5354.956182] Lustre: server umount lustre-OST0000 complete [ 5358.559924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5358.564147] LustreError: Skipped 1 previous similar message [ 5358.566421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5358.570754] Lustre: Skipped 1 previous similar message [ 5362.394056] Lustre: server umount lustre-MDT0000 complete [ 5363.669448] LustreError: 369617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778844315 with bad export cookie 8236595677230402154 [ 5363.671859] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5363.675036] LustreError: 369617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5363.800870] Lustre: server umount lustre-MDT0001 complete [ 5367.456664] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5368.477604] Key type lgssc unregistered [ 5368.600494] LNet: 373808:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5368.603317] LNetError: 373808:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5368.611406] LNet: Removed LNI 192.168.204.126@tcp [ 5368.955597] Key type .llcrypt unregistered [ 5368.957131] Key type ._llcrypt unregistered [ 5375.226633] Key type ._llcrypt registered [ 5375.227994] Key type .llcrypt registered [ 5375.273362] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 5381.468354] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5381.979827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5382.000939] alg: No test for adler32 (adler32-zlib) [ 5382.868975] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 5382.959846] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 5384.543196] Key type lgssc registered [ 5384.919648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5396.413459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5396.417784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5397.508618] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5397.520192] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5397.556762] Lustre: lustre-MDT0000: new disk, initializing [ 5397.578595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5397.584395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5398.777620] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5402.694000] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5402.733766] Lustre: 376903:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5402.759708] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5402.761953] Lustre: Skipped 1 previous similar message [ 5402.798385] Lustre: lustre-MDT0001: new disk, initializing [ 5402.830557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5402.843491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5402.846801] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5404.066570] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5407.168326] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5408.420001] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5410.616747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5410.720749] Lustre: lustre-OST0000: new disk, initializing [ 5410.722231] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5410.724432] Lustre: 378644:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5410.747856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5412.591639] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5413.116660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5413.120573] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5413.152586] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5415.756071] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5417.247455] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5417.340402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5418.745937] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5418.831412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5423.587383] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5423.587481] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5423.588457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5423.589835] LustreError: Skipped 1 previous similar message [ 5423.596543] Lustre: Skipped 1 previous similar message [ 5425.621879] Lustre: server umount lustre-OST0000 complete [ 5428.704948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5428.705495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5428.705947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5428.705952] Lustre: Skipped 1 previous similar message [ 5428.709197] Lustre: Skipped 1 previous similar message [ 5432.992980] Lustre: server umount lustre-MDT0000 complete [ 5434.245274] LustreError: 376894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778844385 with bad export cookie 7742408967479355995 [ 5434.248689] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5434.249137] LustreError: 376894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5434.367267] Lustre: server umount lustre-MDT0001 complete [ 5437.323754] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5437.889920] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 07:26:28 (1778844388) [ 5440.242636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5440.407302] LustreError: 380950:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5440.416721] LustreError: 380950:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 5440.432569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5441.718242] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5444.567555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5444.690728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5445.973700] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5447.717193] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5448.902865] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5451.114043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5453.130694] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5454.938889] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5455.565339] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5471.506219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5475.807863] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5475.810045] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5475.813784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5475.816331] Lustre: Skipped 1 previous similar message [ 5479.123536] Lustre: server umount lustre-OST0000 complete [ 5481.952694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5481.953438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5481.956238] Lustre: Skipped 2 previous similar messages [ 5481.958035] Lustre: Skipped 2 previous similar messages [ 5486.553269] Lustre: server umount lustre-MDT0000 complete [ 5487.071924] LustreError: 382309:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5487.078105] LustreError: 382309:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5487.823127] LustreError: 380932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778844439 with bad export cookie 7742408967479356863 [ 5487.826493] LustreError: 380932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5487.826547] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5487.949635] Lustre: server umount lustre-MDT0001 complete [ 5491.485817] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5492.571383] Key type lgssc unregistered [ 5492.696372] LNet: 384338:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5492.699919] LNetError: 384338:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5492.710360] LNet: Removed LNI 192.168.204.126@tcp [ 5493.018112] Key type .llcrypt unregistered [ 5493.019181] Key type ._llcrypt unregistered [ 5500.644240] Key type ._llcrypt registered [ 5500.645567] Key type .llcrypt registered [ 5500.694473] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5500.798220] Key type .llcrypt unregistered [ 5500.799514] Key type ._llcrypt unregistered [ 5507.581808] Key type ._llcrypt registered [ 5507.583042] Key type .llcrypt registered [ 5507.626330] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5507.982104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5508.034680] alg: No test for adler32 (adler32-zlib) [ 5508.890701] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 5508.985095] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 5510.567176] Key type lgssc registered [ 5510.922254] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5513.528166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5513.534051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5514.719557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5515.992375] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5518.918424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5519.048685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5520.949760] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5522.717619] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5524.448191] LustreError: 386493:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5524.453092] LustreError: 386493:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 5525.168252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5525.318487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5526.375111] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5526.734123] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5530.592781] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5530.595027] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5530.599122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5531.619073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5531.619865] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5531.626617] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5535.766315] Lustre: server umount lustre-OST0000 complete [ 5539.809539] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5539.812502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5539.816923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5539.819643] Lustre: Skipped 1 previous similar message [ 5541.856164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5541.856622] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5543.188306] Lustre: server umount lustre-MDT0000 complete [ 5544.448172] LustreError: 388102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778844495 with bad export cookie 10718327107560989680 [ 5544.451437] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5544.452724] LustreError: 388102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5544.563588] Lustre: server umount lustre-MDT0001 complete [ 5548.236834] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5549.336716] Key type lgssc unregistered [ 5549.464467] LNet: 389471:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5549.467036] LNetError: 389471:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5549.479493] LNet: Removed LNI 192.168.204.126@tcp [ 5549.777737] Key type .llcrypt unregistered [ 5549.779874] Key type ._llcrypt unregistered [ 5558.714796] Key type ._llcrypt registered [ 5558.716538] Key type .llcrypt registered [ 5558.757689] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5559.206619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5559.221933] alg: No test for adler32 (adler32-zlib) [ 5560.089247] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 5560.177715] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 5561.767195] Key type lgssc registered [ 5562.177277] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5564.781170] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5564.786367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5580.255340] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5585.375180] LustreError: 390821:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.126@tcp: failed processing log, type 4: rc = -110 [ 5611.999233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5614.041216] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5615.801295] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5618.131290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5618.326216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5618.363328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5619.666729] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5622.690693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5622.836679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5624.237278] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5626.094679] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5627.308327] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5631.970074] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5631.976141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5631.978451] Lustre: Skipped 1 previous similar message [ 5635.728314] Lustre: server umount lustre-OST0000 complete [ 5637.088669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5637.089575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5637.092440] Lustre: Skipped 2 previous similar messages [ 5637.096483] Lustre: Skipped 1 previous similar message [ 5642.208231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5642.211275] Lustre: Skipped 1 previous similar message [ 5643.096418] Lustre: server umount lustre-MDT0000 complete [ 5644.357257] LustreError: 392455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778844595 with bad export cookie 841202358981776442 [ 5644.360788] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5644.361034] LustreError: 392455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5644.489202] Lustre: server umount lustre-MDT0001 complete [ 5648.301868] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5649.331042] Key type lgssc unregistered [ 5649.462441] LNet: 394032:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5649.464689] LNetError: 394032:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5649.472302] LNet: Removed LNI 192.168.204.126@tcp [ 5649.766181] Key type .llcrypt unregistered [ 5649.767173] Key type ._llcrypt unregistered [ 5658.472987] Key type ._llcrypt registered [ 5658.474056] Key type .llcrypt registered [ 5658.512984] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5658.885509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5658.954310] alg: No test for adler32 (adler32-zlib) [ 5659.804528] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 5659.880685] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 5661.455181] Key type lgssc registered [ 5661.892338] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5664.752601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5664.759606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5665.944222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5667.276267] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5670.174546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5670.304365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5671.644141] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5673.463668] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5674.641297] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5676.828764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5676.948700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5678.864185] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5680.574877] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5691.303039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5701.087933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5701.091178] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5701.096244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5701.601552] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5701.602511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5701.605609] Lustre: Skipped 1 previous similar message [ 5701.607276] Lustre: Skipped 1 previous similar message [ 5703.260880] Lustre: server umount lustre-MDT0000 complete [ 5704.042838] LustreError: 397021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.26@tcp arrived at 1778844655 with bad export cookie 13580261631964480620 [ 5704.047559] LustreError: 397021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5704.697459] LustreError: 397668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778844656 with bad export cookie 13580261631964479969 [ 5704.701585] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5705.695867] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5705.698451] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5705.703395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5705.706809] Lustre: Skipped 1 previous similar message [ 5706.719860] LustreError: 395407:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5706.724983] LustreError: 395407:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 5709.727844] LustreError: 395408:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5709.728177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5709.732765] LustreError: 395408:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5709.735235] Lustre: Skipped 1 previous similar message [ 5710.800617] Lustre: server umount lustre-OST0000 complete [ 5713.112310] Lustre: server umount lustre-MDT0001 complete [ 5716.818536] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5717.910306] Key type lgssc unregistered [ 5718.043522] LNet: 398754:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5718.046634] LNetError: 398754:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5718.055401] LNet: Removed LNI 192.168.204.126@tcp [ 5718.368608] Key type .llcrypt unregistered [ 5718.370256] Key type ._llcrypt unregistered [ 5727.344950] Key type ._llcrypt registered [ 5727.346015] Key type .llcrypt registered [ 5727.379635] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5727.690582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5727.740115] alg: No test for adler32 (adler32-zlib) [ 5728.588262] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 5728.674879] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 5730.255143] Key type lgssc registered [ 5730.562507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5733.088378] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5733.094050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5734.274733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5735.637833] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5738.555427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5738.682322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5740.007628] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5741.878344] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5743.193304] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5745.502368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5745.619946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5747.591388] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5749.295131] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5758.949823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5764.065120] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5764.069131] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5764.070799] Lustre: Skipped 1 previous similar message [ 5764.820142] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.26@tcp (stopping) [ 5766.740730] Lustre: server umount lustre-MDT0001 complete [ 5769.184767] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5769.185045] LustreError: 400130:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5769.185143] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5769.185150] Lustre: Skipped 1 previous similar message [ 5769.186124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5769.203858] LustreError: 400130:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5774.304129] LustreError: 400131:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5774.304693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5774.480192] Lustre: server umount lustre-OST0000 complete [ 5775.842288] Lustre: server umount lustre-MDT0000 complete [ 5780.252835] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5781.343092] Key type lgssc unregistered [ 5781.470427] LNet: 403477:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5781.473263] LNetError: 403477:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5781.480334] LNet: Removed LNI 192.168.204.126@tcp [ 5781.771908] Key type .llcrypt unregistered [ 5781.773092] Key type ._llcrypt unregistered [ 5789.685492] Key type ._llcrypt registered [ 5789.686545] Key type .llcrypt registered [ 5789.722693] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 5795.648939] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5795.986749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5796.041954] alg: No test for adler32 (adler32-zlib) [ 5796.893285] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 5796.975432] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 5798.559136] Key type lgssc registered [ 5798.920290] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5811.075725] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5816.328246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5816.334025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5817.451715] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5817.463266] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5817.498265] Lustre: lustre-MDT0000: new disk, initializing [ 5817.537068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5817.542367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5818.813894] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5824.124959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5824.227465] Lustre: lustre-OST0000: new disk, initializing [ 5824.229038] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5824.230845] Lustre: Skipped 1 previous similar message [ 5824.232547] Lustre: 408412:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5824.256521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5826.188474] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5827.133499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 5827.136366] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 5827.146302] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 5829.297213] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5834.791969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5834.840469] Lustre: 407412:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5834.872615] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5834.912386] Lustre: lustre-MDT0001: new disk, initializing [ 5834.953317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5834.967874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 5834.971121] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 5836.308147] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5839.884598] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5839.959948] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5841.269934] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5843.387931] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 2 sec [ 5847.523211] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5847.523339] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5847.525546] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5847.528060] Lustre: Skipped 1 previous similar message [ 5848.787540] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.26@tcp (stopping) [ 5848.790307] Lustre: Skipped 1 previous similar message [ 5851.221112] Lustre: server umount lustre-MDT0001 complete [ 5857.762876] LustreError: 410375:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5857.767929] LustreError: 410375:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 5862.882396] LustreError: 408851:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5862.887133] LustreError: 408851:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5868.001622] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5868.002541] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5868.002600] LustreError: 410375:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5868.014337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5869.588603] Lustre: server umount lustre-OST0000 complete [ 5885.407141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5885.501932] Lustre: server umount lustre-MDT0000 complete [ 5890.078644] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5891.120111] Key type lgssc unregistered [ 5891.260370] LNet: 411990:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5891.263515] LNetError: 411990:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5891.275344] LNet: Removed LNI 192.168.204.126@tcp [ 5891.582144] Key type .llcrypt unregistered [ 5891.583446] Key type ._llcrypt unregistered [ 5904.463991] Key type ._llcrypt registered [ 5904.465064] Key type .llcrypt registered [ 5904.503233] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5904.613246] Key type .llcrypt unregistered [ 5904.614990] Key type ._llcrypt unregistered [ 5911.347154] Key type ._llcrypt registered [ 5911.348826] Key type .llcrypt registered [ 5911.386296] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5911.755732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5911.783063] alg: No test for adler32 (adler32-zlib) [ 5912.638855] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 5912.727386] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 5914.311241] Key type lgssc registered [ 5914.725801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5923.648949] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5923.656511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5924.763159] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5924.775486] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5924.818240] Lustre: lustre-MDT0000: new disk, initializing [ 5924.847558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5924.855969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5926.187512] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5930.394666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5930.436904] Lustre: 415307:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5930.461927] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5930.463836] Lustre: Skipped 1 previous similar message [ 5930.499582] Lustre: lustre-MDT0001: new disk, initializing [ 5930.536048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5930.548159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5930.553468] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5931.967385] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5935.119493] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5936.290107] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5938.578529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5938.695895] Lustre: lustre-OST0000: new disk, initializing [ 5938.697481] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5938.699909] Lustre: 417047:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5938.730109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5940.601969] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5942.522042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5942.526630] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5942.561775] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5943.743576] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5947.876953] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5947.879097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5947.881363] Lustre: Skipped 2 previous similar messages [ 5947.885461] Lustre: Skipped 2 previous similar messages [ 5952.290864] Lustre: server umount lustre-MDT0000 complete [ 5953.552291] LustreError: 416084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778844904 with bad export cookie 12872137642469642853 [ 5953.555597] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5953.556479] LustreError: 416084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5953.682092] Lustre: server umount lustre-MDT0001 complete [ 5965.219852] Lustre: server umount lustre-OST0000 complete [ 5968.864421] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 07:35:19 (1778844919) [ 5971.538581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5971.711187] LustreError: 418958:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5971.717162] LustreError: 418958:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 5971.734205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5973.029434] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5975.833042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5975.960199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5977.200573] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5978.949312] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5980.148943] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5995.490932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5995.574510] Lustre: Found index 0 for lustre-OST0000, updating log [ 5995.643703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5996.711409] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 5997.519847] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5999.202466] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6001.314193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6001.315419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6001.320591] Lustre: Skipped 1 previous similar message [ 6006.411359] Lustre: server umount lustre-OST0000 complete [ 6022.111988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6022.115833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6022.122225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6024.160021] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6024.162980] Lustre: Skipped 3 previous similar messages [ 6029.280168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6029.283075] Lustre: Skipped 1 previous similar message [ 6033.375102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6033.439513] Lustre: server umount lustre-MDT0000 complete [ 6034.400067] LustreError: 418959:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6034.407681] LustreError: 418959:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 6034.705301] LustreError: 420574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778844986 with bad export cookie 12872137642469644232 [ 6034.708093] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6034.710018] LustreError: 420574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6034.831690] Lustre: server umount lustre-MDT0001 complete [ 6037.733316] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 07:36:28 (1778844988) [ 6061.215227] LNet: 1 local NIs in recovery (showing 1): 192.168.204.126@tcp [ 6617.082187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6617.233070] LustreError: 425018:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6617.255242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6618.618649] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6621.432971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6622.802427] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6624.571490] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6625.758948] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6640.994572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6641.096268] Lustre: Found index 0 for lustre-OST0000, updating log [ 6642.994384] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6644.262250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 6644.740646] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6647.457080] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6647.461123] Lustre: Skipped 1 previous similar message [ 6647.462922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6648.481187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6650.530447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6650.532891] Lustre: Skipped 1 previous similar message [ 6651.531160] Lustre: server umount lustre-OST0000 complete [ 6653.792654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6653.795877] Lustre: Skipped 1 previous similar message [ 6654.816279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6654.819390] Lustre: Skipped 3 previous similar messages [ 6663.008140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6663.011400] Lustre: Skipped 15 previous similar messages [ 6667.231101] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6667.290304] Lustre: server umount lustre-MDT0000 complete [ 6668.128181] LustreError: 425732:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6668.132229] LustreError: 425732:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 6668.880546] LustreError: 424999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778845620 with bad export cookie 12872137642469645289 [ 6668.885952] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6668.889710] LustreError: 424999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6669.042923] Lustre: server umount lustre-MDT0001 complete [ 6675.615248] LNet: 1 local NIs in recovery (showing 1): 192.168.204.126@tcp [ 6686.416486] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 6689.936519] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 6702.003220] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 07:47:32 (1778845652) [ 6708.175350] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 6709.368192] Key type lgssc unregistered [ 6709.507548] LNet: 431283:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6709.511375] LNetError: 431283:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6772.005293] LNet: Removed LNI 192.168.204.126@tcp [ 6772.289423] Key type .llcrypt unregistered [ 6772.290563] Key type ._llcrypt unregistered [ 6778.668164] Key type ._llcrypt registered [ 6778.669089] Key type .llcrypt registered [ 6778.704042] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 6779.090413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6779.098799] alg: No test for adler32 (adler32-zlib) [ 6779.948878] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 6780.025127] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 6781.607191] Key type lgssc registered [ 6781.950645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6783.297400] LNet: Added LNI 192.168.204.126@tcp100 [8/256/0/180] [ 6783.590280] LNet: Added LNI 192.168.204.126@tcp101 [8/256/0/180] [ 6783.863547] LNet: Added LNI 192.168.204.126@tcp102 [8/256/0/180] [ 6784.182701] LNet: Added LNI 192.168.204.126@tcp103 [8/256/0/180] [ 6784.556947] LNet: Added LNI 192.168.204.126@tcp104 [8/256/0/180] [ 6784.870090] LNet: Added LNI 192.168.204.126@tcp105 [8/256/0/180] [ 6785.170342] LNet: Added LNI 192.168.204.126@tcp106 [8/256/0/180] [ 6785.456611] LNet: Added LNI 192.168.204.126@tcp107 [8/256/0/180] [ 6785.743755] LNet: Added LNI 192.168.204.126@tcp108 [8/256/0/180] [ 6786.023932] LNet: Added LNI 192.168.204.126@tcp109 [8/256/0/180] [ 6786.315855] LNet: Added LNI 192.168.204.126@tcp110 [8/256/0/180] [ 6786.620063] LNet: Added LNI 192.168.204.126@tcp111 [8/256/0/180] [ 6786.947191] LNet: Added LNI 192.168.204.126@tcp112 [8/256/0/180] [ 6787.265501] LNet: Added LNI 192.168.204.126@tcp113 [8/256/0/180] [ 6787.554676] LNet: Added LNI 192.168.204.126@tcp114 [8/256/0/180] [ 6787.848786] LNet: Added LNI 192.168.204.126@tcp115 [8/256/0/180] [ 6788.193564] LNet: Added LNI 192.168.204.126@tcp116 [8/256/0/180] [ 6788.495067] LNet: Added LNI 192.168.204.126@tcp117 [8/256/0/180] [ 6788.783264] LNet: Added LNI 192.168.204.126@tcp118 [8/256/0/180] [ 6789.075691] LNet: Added LNI 192.168.204.126@tcp119 [8/256/0/180] [ 6789.368029] LNet: Added LNI 192.168.204.126@tcp120 [8/256/0/180] [ 6789.658801] LNet: Added LNI 192.168.204.126@tcp121 [8/256/0/180] [ 6789.949461] LNet: Added LNI 192.168.204.126@tcp122 [8/256/0/180] [ 6790.236662] LNet: Added LNI 192.168.204.126@tcp123 [8/256/0/180] [ 6790.521344] LNet: Added LNI 192.168.204.126@tcp124 [8/256/0/180] [ 6790.813592] LNet: Added LNI 192.168.204.126@tcp125 [8/256/0/180] [ 6791.095234] LNet: Added LNI 192.168.204.126@tcp126 [8/256/0/180] [ 6791.363463] LNet: Added LNI 192.168.204.126@tcp127 [8/256/0/180] [ 6791.652832] LNet: Added LNI 192.168.204.126@tcp128 [8/256/0/180] [ 6791.945294] LNet: Added LNI 192.168.204.126@tcp129 [8/256/0/180] [ 6792.237630] LNet: Added LNI 192.168.204.126@tcp130 [8/256/0/180] [ 6792.521579] LNet: Added LNI 192.168.204.126@tcp131 [8/256/0/180] [ 6792.815955] LNet: Added LNI 192.168.204.126@tcp132 [8/256/0/180] [ 6793.110261] LNet: Added LNI 192.168.204.126@tcp133 [8/256/0/180] [ 6793.395778] LNet: Added LNI 192.168.204.126@tcp134 [8/256/0/180] [ 6793.692794] LNet: Added LNI 192.168.204.126@tcp135 [8/256/0/180] [ 6795.834633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6795.839778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6796.881580] Lustre: 434263:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.126@tcp: can't add NID 192.168.204.126@tcp130: rc = -75 [ 6796.913778] LNetError: 431729:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [ 6796.917508] LNetError: Protocol error connecting to 192.168.204.126@tcp118 at host 192.168.204.126:988: is it running a compatible version of Lustre? [ 6796.954541] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6796.964746] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6797.000249] Lustre: lustre-MDT0000: new disk, initializing [ 6797.029686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6797.035342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6797.471175] LNet: 1 local NIs in recovery (showing 1): 192.168.204.126@tcp118 [ 6797.471888] LNetError: 431728:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [ 6797.475717] LNetError: Protocol error connecting to 192.168.204.126@tcp102 at host 192.168.204.126:988: is it running a compatible version of Lustre? [ 6798.276381] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6799.520914] LNetError: 431728:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [ 6799.526402] LNetError: 431728:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [ 6799.531143] LNetError: Protocol error connecting to 192.168.204.126@tcp118 at host 192.168.204.126:988: is it running a compatible version of Lustre? [ 6799.534377] LNetError: Skipped 3 previous similar messages [ 6802.346067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6802.390496] Lustre: 434281:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6802.414391] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6802.416422] Lustre: Skipped 1 previous similar message [ 6802.453372] Lustre: lustre-MDT0001: new disk, initializing [ 6802.466249] Lustre: 435066:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.126@tcp: can't add NID 192.168.204.126@tcp130: rc = -75 [ 6802.469062] Lustre: 435066:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 6802.508078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6802.520996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6802.524130] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6803.617340] LNetError: 434291:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [ 6803.620797] LNetError: Protocol error connecting to 192.168.204.126@tcp118 at host 192.168.204.126:988: is it running a compatible version of Lustre? [ 6803.782535] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6805.361539] Lustre: 435413:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.126@tcp: can't add NID 192.168.204.126@tcp130: rc = -75 [ 6805.366096] Lustre: 435413:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 6806.793366] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6807.968475] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6810.190176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6810.296972] Lustre: lustre-OST0000: new disk, initializing [ 6810.298565] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6810.300923] Lustre: 436027:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6810.332406] Lustre: 436032:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.126@tcp: can't add NID 192.168.204.126@tcp130: rc = -75 [ 6810.335175] Lustre: 436032:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 6810.352039] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6811.808170] LNetError: 434290:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [ 6811.811655] LNetError: Protocol error connecting to 192.168.204.126@tcp118 at host 192.168.204.126:988: is it running a compatible version of Lustre? [ 6812.153198] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6813.757291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6813.760775] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6813.800974] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6815.196143] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6818.784056] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6818.787207] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6818.791970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6819.298310] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6819.298449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6819.301985] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6823.566406] Lustre: server umount lustre-OST0000 complete [ 6827.168146] LNetError: 434285:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [ 6827.171032] LNetError: Protocol error connecting to 192.168.204.126@tcp118 at host 192.168.204.126:988: is it running a compatible version of Lustre? [ 6827.999882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6828.003827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6828.010169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6828.012247] Lustre: Skipped 1 previous similar message [ 6834.656113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6834.658316] Lustre: Skipped 3 previous similar messages [ 6839.263093] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6839.321670] Lustre: server umount lustre-MDT0000 complete [ 6840.588363] LustreError: 435065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778845792 with bad export cookie 11293604904293447050 [ 6840.591962] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6840.592271] LustreError: 435065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6840.713819] Lustre: server umount lustre-MDT0001 complete [ 6844.211147] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 6845.197548] Key type lgssc unregistered [ 6845.358574] LNet: 437808:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6845.361596] LNetError: 437808:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6845.363824] LNet: Removed LNI 192.168.204.126@tcp [ 6846.367213] LNet: Removed LNI 192.168.204.126@tcp100 [ 6846.368862] LNetError: 437808:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6846.371007] LNetError: 437808:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 6846.373014] LNet: Removed LNI 192.168.204.126@tcp101 [ 6847.391146] LNet: Removed LNI 192.168.204.126@tcp102 [ 6847.393101] LNetError: 437808:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6847.396327] LNetError: 437808:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 6847.400298] LNet: Removed LNI 192.168.204.126@tcp103 [ 6848.415147] LNet: Removed LNI 192.168.204.126@tcp104 [ 6849.439100] LNet: Removed LNI 192.168.204.126@tcp105 [ 6849.440451] LNetError: 437808:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6849.443063] LNetError: 437808:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 6849.446574] LNet: Removed LNI 192.168.204.126@tcp106 [ 6850.463125] LNet: Removed LNI 192.168.204.126@tcp107 [ 6851.487153] LNet: Removed LNI 192.168.204.126@tcp108 [ 6852.511133] LNet: Removed LNI 192.168.204.126@tcp109 [ 6853.535171] LNet: Removed LNI 192.168.204.126@tcp110 [ 6853.536561] LNetError: 437808:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6853.539470] LNetError: 437808:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 6853.542263] LNet: Removed LNI 192.168.204.126@tcp111 [ 6854.559098] LNet: Removed LNI 192.168.204.126@tcp112 [ 6855.583171] LNet: Removed LNI 192.168.204.126@tcp113 [ 6856.607187] LNet: Removed LNI 192.168.204.126@tcp114 [ 6857.631131] LNet: Removed LNI 192.168.204.126@tcp115 [ 6858.655134] LNet: Removed LNI 192.168.204.126@tcp116 [ 6859.679123] LNet: Removed LNI 192.168.204.126@tcp117 [ 6859.680935] LNet: Removed LNI 192.168.204.126@tcp118 [ 6859.682655] LNet: Removed LNI 192.168.204.126@tcp119 [ 6859.684321] LNet: Removed LNI 192.168.204.126@tcp120 [ 6859.686034] LNet: Removed LNI 192.168.204.126@tcp121 [ 6859.687719] LNet: Removed LNI 192.168.204.126@tcp122 [ 6859.689389] LNet: Removed LNI 192.168.204.126@tcp123 [ 6859.691103] LNet: Removed LNI 192.168.204.126@tcp124 [ 6859.692756] LNet: Removed LNI 192.168.204.126@tcp125 [ 6859.694414] LNet: Removed LNI 192.168.204.126@tcp126 [ 6859.696094] LNet: Removed LNI 192.168.204.126@tcp127 [ 6859.697765] LNet: Removed LNI 192.168.204.126@tcp128 [ 6859.699422] LNet: Removed LNI 192.168.204.126@tcp129 [ 6859.701070] LNet: Removed LNI 192.168.204.126@tcp130 [ 6859.702756] LNet: Removed LNI 192.168.204.126@tcp131 [ 6859.704398] LNet: Removed LNI 192.168.204.126@tcp132 [ 6859.706053] LNet: Removed LNI 192.168.204.126@tcp133 [ 6859.707718] LNet: Removed LNI 192.168.204.126@tcp134 [ 6859.715460] LNet: Removed LNI 192.168.204.126@tcp135 [ 6859.965793] Key type .llcrypt unregistered [ 6859.966834] Key type ._llcrypt unregistered [ 6867.103241] Key type ._llcrypt registered [ 6867.104534] Key type .llcrypt registered [ 6867.144616] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 6873.068710] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 6873.513272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6873.523168] alg: No test for adler32 (adler32-zlib) [ 6874.372709] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 6874.456591] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 6876.039168] Key type lgssc registered [ 6876.406774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6888.004497] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 07:50:38 (1778845838) [ 6894.309237] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 6895.445077] Key type lgssc unregistered [ 6895.582406] LNet: 441814:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6895.585440] LNetError: 441814:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6895.599354] LNet: Removed LNI 192.168.204.126@tcp [ 6895.900538] Key type .llcrypt unregistered [ 6895.901933] Key type ._llcrypt unregistered [ 6902.703465] Key type ._llcrypt registered [ 6902.704828] Key type .llcrypt registered [ 6902.746057] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 6903.094611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6903.107797] alg: No test for adler32 (adler32-zlib) [ 6903.993286] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 6904.087460] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 6905.671179] Key type lgssc registered [ 6906.046213] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6909.684342] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6909.690149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6910.800681] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6910.812479] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6910.852611] Lustre: lustre-MDT0000: new disk, initializing [ 6910.880710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6910.887588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6912.218119] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6916.383793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6916.432590] Lustre: 442987:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6916.463982] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6916.467962] Lustre: Skipped 1 previous similar message [ 6916.511459] Lustre: lustre-MDT0001: new disk, initializing [ 6916.548149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6916.562775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6916.567130] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6918.147620] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6921.567769] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6922.740690] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6925.162202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6925.288546] Lustre: lustre-OST0000: new disk, initializing [ 6925.290165] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6925.292957] Lustre: 444729:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6925.331697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6925.792658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6925.796916] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6925.832592] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6927.490688] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6930.577750] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6931.760673] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing lnet_if_list [ 6933.392549] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 6933.403042] LNet: Added LNI 192.168.204.126@tcp10 [8/256/0/180] [ 6935.111246] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 6935.123781] LNet: Added LNI 192.168.204.126@tcp11 [8/256/0/180] [ 6936.385512] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 6936.396662] LNet: Added LNI 192.168.204.126@tcp12 [8/256/0/180] [ 6937.671560] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 6937.683888] LNet: Added LNI 192.168.204.126@tcp13 [8/256/0/180] [ 6938.975608] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 6938.985967] LNet: Added LNI 192.168.204.126@tcp14 [8/256/0/180] [ 6940.358711] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 6940.370822] LNet: Added LNI 192.168.204.126@tcp15 [8/256/0/180] [ 6941.807768] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 6941.818735] LNet: Added LNI 192.168.204.126@tcp16 [8/256/0/180] [ 6943.319142] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 6943.331091] LNet: Added LNI 192.168.204.126@tcp17 [8/256/0/180] [ 6944.808800] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 6944.818563] LNet: Added LNI 192.168.204.126@tcp18 [8/256/0/180] [ 6946.301227] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 6946.311228] LNet: Added LNI 192.168.204.126@tcp19 [8/256/0/180] [ 6947.883945] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 6947.894567] LNet: Added LNI 192.168.204.126@tcp20 [8/256/0/180] [ 6949.489556] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 6949.501964] LNet: Added LNI 192.168.204.126@tcp21 [8/256/0/180] [ 6951.068014] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 6951.081381] LNet: Added LNI 192.168.204.126@tcp22 [8/256/0/180] [ 6952.651828] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 6952.663108] LNet: Added LNI 192.168.204.126@tcp23 [8/256/0/180] [ 6954.348850] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 6954.359373] LNet: Added LNI 192.168.204.126@tcp24 [8/256/0/180] [ 6956.063070] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 6956.075540] LNet: Added LNI 192.168.204.126@tcp25 [8/256/0/180] [ 6957.786473] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 6957.801480] LNet: Added LNI 192.168.204.126@tcp26 [8/256/0/180] [ 6959.537017] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 6959.549525] LNet: Added LNI 192.168.204.126@tcp27 [8/256/0/180] [ 6961.315718] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 6961.325321] LNet: Added LNI 192.168.204.126@tcp28 [8/256/0/180] [ 6963.204235] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 6963.214286] LNet: Added LNI 192.168.204.126@tcp29 [8/256/0/180] [ 6965.033448] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 6965.046485] LNet: Added LNI 192.168.204.126@tcp30 [8/256/0/180] [ 6967.080619] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 6967.093731] LNet: Added LNI 192.168.204.126@tcp31 [8/256/0/180] [ 6969.149241] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 6969.164557] LNet: Added LNI 192.168.204.126@tcp32 [8/256/0/180] [ 6971.230303] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 6971.240921] LNet: Added LNI 192.168.204.126@tcp33 [8/256/0/180] [ 6973.418579] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 6973.431068] LNet: Added LNI 192.168.204.126@tcp34 [8/256/0/180] [ 6975.662842] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 6975.675176] LNet: Added LNI 192.168.204.126@tcp35 [8/256/0/180] [ 6977.753221] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 6977.768390] LNet: Added LNI 192.168.204.126@tcp36 [8/256/0/180] [ 6980.136701] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 6980.148032] LNet: Added LNI 192.168.204.126@tcp37 [8/256/0/180] [ 6982.410517] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 6982.420653] LNet: Added LNI 192.168.204.126@tcp38 [8/256/0/180] [ 6984.719870] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 6984.730334] LNet: Added LNI 192.168.204.126@tcp39 [8/256/0/180] [ 6987.130788] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 6987.142308] LNet: Added LNI 192.168.204.126@tcp40 [8/256/0/180] [ 6989.489141] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 6989.501120] LNet: Added LNI 192.168.204.126@tcp41 [8/256/0/180] [ 6991.952945] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 6991.964718] LNet: Added LNI 192.168.204.126@tcp42 [8/256/0/180] [ 6994.302821] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 6994.314852] LNet: Added LNI 192.168.204.126@tcp43 [8/256/0/180] [ 6996.600748] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 6996.610396] LNet: Added LNI 192.168.204.126@tcp44 [8/256/0/180] [ 6998.975208] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 6998.989720] LNet: Added LNI 192.168.204.126@tcp45 [8/256/0/180] [ 7001.349624] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7001.360564] LNet: Added LNI 192.168.204.126@tcp46 [8/256/0/180] [ 7003.729635] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7003.742477] LNet: Added LNI 192.168.204.126@tcp47 [8/256/0/180] [ 7006.285121] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7006.296436] LNet: Added LNI 192.168.204.126@tcp48 [8/256/0/180] [ 7008.774853] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7008.785427] LNet: Added LNI 192.168.204.126@tcp49 [8/256/0/180] [ 7011.333554] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7011.346707] LNet: Added LNI 192.168.204.126@tcp50 [8/256/0/180] [ 7018.976389] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7018.981548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7018.989913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7023.074962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7023.076123] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7023.079615] Lustre: Skipped 1 previous similar message [ 7023.087604] Lustre: Skipped 1 previous similar message [ 7024.803309] Lustre: server umount lustre-MDT0000 complete [ 7026.072139] LustreError: 442980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778845977 with bad export cookie 17892518690034829612 [ 7026.076576] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7026.078589] LustreError: 442980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7026.206971] Lustre: server umount lustre-MDT0001 complete [ 7033.367732] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7036.754958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7044.511112] Lustre: 442260:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778845979/real 1778845979] req@000000005069cd51 x1865255071332736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778845995 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7044.520237] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7046.623094] Lustre: 442259:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778845982/real 1778845982] req@00000000a5d09b93 x1865255071332864/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778845998 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7048.671102] Lustre: 442261:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778845984/real 1778845984] req@000000001d073443 x1865255071332992/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778846000 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7050.655476] LustreError: 442258:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000092fefeb4 x1865255071333888/t0(0) o250->MGC192.168.204.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7050.854459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7051.871089] Lustre: 442262:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778845987/real 1778845987] req@00000000076f10a6 x1865255071333248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778846003 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7052.231809] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7055.089294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7055.214715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7056.503507] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7056.864097] Lustre: 442259:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778845992/real 1778845992] req@0000000035763421 x1865255071333376/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778846008 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7058.313309] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7059.605939] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7060.452850] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7063.525780] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7068.642329] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7068.643167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7068.645907] Lustre: Skipped 1 previous similar message [ 7068.651664] Lustre: Skipped 1 previous similar message [ 7073.676661] Lustre: server umount lustre-OST0000 complete [ 7075.807875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7075.810525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7075.814564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7081.046792] Lustre: server umount lustre-MDT0000 complete [ 7082.262094] LustreError: 444728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778846033 with bad export cookie 17892518690034831614 [ 7082.265573] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7082.268441] LustreError: 444728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7082.371482] Lustre: server umount lustre-MDT0001 complete [ 7085.780588] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7086.758809] Key type lgssc unregistered [ 7086.917571] LNet: 470601:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7086.922272] LNetError: 470601:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7086.924681] LNet: Removed LNI 192.168.204.126@tcp [ 7086.926111] LNet: Removed LNI 192.168.204.126@tcp10 [ 7086.928059] LNet: Removed LNI 192.168.204.126@tcp11 [ 7086.930298] LNet: Removed LNI 192.168.204.126@tcp12 [ 7086.932503] LNet: Removed LNI 192.168.204.126@tcp13 [ 7086.934732] LNet: Removed LNI 192.168.204.126@tcp14 [ 7086.936313] LNet: Removed LNI 192.168.204.126@tcp15 [ 7086.937490] LNet: Removed LNI 192.168.204.126@tcp16 [ 7086.938647] LNet: Removed LNI 192.168.204.126@tcp17 [ 7086.939767] LNet: Removed LNI 192.168.204.126@tcp18 [ 7086.940869] LNet: Removed LNI 192.168.204.126@tcp19 [ 7086.941963] LNet: Removed LNI 192.168.204.126@tcp20 [ 7086.943087] LNet: Removed LNI 192.168.204.126@tcp21 [ 7086.944189] LNet: Removed LNI 192.168.204.126@tcp22 [ 7086.945292] LNet: Removed LNI 192.168.204.126@tcp23 [ 7086.946362] LNet: Removed LNI 192.168.204.126@tcp24 [ 7086.947436] LNet: Removed LNI 192.168.204.126@tcp25 [ 7086.948578] LNet: Removed LNI 192.168.204.126@tcp26 [ 7086.949667] LNet: Removed LNI 192.168.204.126@tcp27 [ 7086.950773] LNet: Removed LNI 192.168.204.126@tcp28 [ 7086.951809] LNet: Removed LNI 192.168.204.126@tcp29 [ 7086.952914] LNet: Removed LNI 192.168.204.126@tcp30 [ 7086.954069] LNet: Removed LNI 192.168.204.126@tcp31 [ 7086.955200] LNet: Removed LNI 192.168.204.126@tcp32 [ 7086.956293] LNet: Removed LNI 192.168.204.126@tcp33 [ 7086.957411] LNet: Removed LNI 192.168.204.126@tcp34 [ 7086.958506] LNet: Removed LNI 192.168.204.126@tcp35 [ 7086.959616] LNet: Removed LNI 192.168.204.126@tcp36 [ 7086.960800] LNet: Removed LNI 192.168.204.126@tcp37 [ 7086.961877] LNet: Removed LNI 192.168.204.126@tcp38 [ 7086.963020] LNet: Removed LNI 192.168.204.126@tcp39 [ 7086.964036] LNet: Removed LNI 192.168.204.126@tcp40 [ 7086.965030] LNet: Removed LNI 192.168.204.126@tcp41 [ 7086.966289] LNet: Removed LNI 192.168.204.126@tcp42 [ 7086.967394] LNet: Removed LNI 192.168.204.126@tcp43 [ 7086.968563] LNet: Removed LNI 192.168.204.126@tcp44 [ 7086.969573] LNet: Removed LNI 192.168.204.126@tcp45 [ 7086.970723] LNet: Removed LNI 192.168.204.126@tcp46 [ 7086.971872] LNet: Removed LNI 192.168.204.126@tcp47 [ 7086.973064] LNet: Removed LNI 192.168.204.126@tcp48 [ 7086.974045] LNet: Removed LNI 192.168.204.126@tcp49 [ 7086.981361] LNet: Removed LNI 192.168.204.126@tcp50 [ 7087.243832] Key type .llcrypt unregistered [ 7087.245155] Key type ._llcrypt unregistered [ 7093.420170] Key type ._llcrypt registered [ 7093.421270] Key type .llcrypt registered [ 7093.454813] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 7099.299445] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7099.623944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7099.687977] alg: No test for adler32 (adler32-zlib) [ 7100.551572] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 7100.631946] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 7102.207156] Key type lgssc registered [ 7102.568749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7113.565858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7113.576753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7114.664296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7114.676062] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7114.714920] Lustre: lustre-MDT0000: new disk, initializing [ 7114.738809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7114.745335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7116.054142] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7120.239186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7120.276977] Lustre: 473735:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7120.299898] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7120.302619] Lustre: Skipped 1 previous similar message [ 7120.339107] Lustre: lustre-MDT0001: new disk, initializing [ 7120.370428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7120.381764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7120.384560] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7121.589301] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7124.622499] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7125.753703] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7128.407082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7128.482281] Lustre: *** cfs_fail_loc=90f, val=0*** [ 7128.510187] Lustre: lustre-OST0000: new disk, initializing [ 7128.512883] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7128.515621] Lustre: 475524:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7128.543778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7130.382301] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7133.827383] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7134.711211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7134.714241] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7134.749477] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7139.809572] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7139.810370] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7139.813101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7139.816804] Lustre: Skipped 1 previous similar message [ 7142.253190] Lustre: server umount lustre-OST0000 complete [ 7144.930755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7144.932323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7144.941368] Lustre: Skipped 1 previous similar message [ 7144.947883] Lustre: Skipped 1 previous similar message [ 7150.048191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7150.050128] Lustre: Skipped 1 previous similar message [ 7155.168236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7155.170472] Lustre: Skipped 1 previous similar message [ 7158.239088] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7158.298861] Lustre: server umount lustre-MDT0000 complete [ 7159.479475] LustreError: 473728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778846110 with bad export cookie 4075912928383656391 [ 7159.482833] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7159.485339] LustreError: 473728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7159.605506] Lustre: server umount lustre-MDT0001 complete [ 7163.111589] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7164.146732] Key type lgssc unregistered [ 7164.277362] LNet: 477298:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7164.279554] LNetError: 477298:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7164.287317] LNet: Removed LNI 192.168.204.126@tcp [ 7164.570206] Key type .llcrypt unregistered [ 7164.572172] Key type ._llcrypt unregistered [ 7171.800172] Key type ._llcrypt registered [ 7171.801182] Key type .llcrypt registered [ 7171.837777] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7172.237649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7172.252265] alg: No test for adler32 (adler32-zlib) [ 7173.104037] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 7173.178655] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 7174.759168] Key type lgssc registered [ 7175.125141] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7177.661249] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7177.667094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7178.819721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7180.081291] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7182.826914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7182.951057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7184.194616] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7185.897708] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7187.037954] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7189.275055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7189.390442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7191.193607] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7192.891811] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7197.477311] loop0: detected capacity change from 0 to 204800000 [ 7197.540839] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7197.545226] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7197.549647] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7199.443813] loop0: detected capacity change from 0 to 204800000 [ 7199.495866] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7199.500877] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7199.504760] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7201.176074] loop0: detected capacity change from 0 to 204800000 [ 7203.032730] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7203.086708] Lustre: fs15246-OST0000: new disk, initializing [ 7203.088793] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7203.091813] Lustre: 481780:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7203.112052] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7205.079190] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7208.944485] loop1: detected capacity change from 0 to 204800000 [ 7210.781195] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7210.820262] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7210.843515] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7210.880538] Lustre: fs15246-MDT0000: new disk, initializing [ 7210.925391] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7210.934262] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7210.966331] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7211.999845] Lustre: 482882:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000df84d11c x1865255353348224/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1778846179 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7212.209673] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7221.087182] LustreError: 482882:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 7221.090097] LustreError: 482882:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 7221.092739] LustreError: 482882:0:(osp_precreate.c:460:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 7223.779742] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7223.783603] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7225.691150] Lustre: server umount fs15246-MDT0000 complete [ 7228.728233] Lustre: server umount fs15246-OST0000 complete [ 7234.018789] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7234.023809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7234.025506] Lustre: Skipped 1 previous similar message [ 7239.138160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7239.140008] Lustre: Skipped 1 previous similar message [ 7244.259407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7244.262924] Lustre: Skipped 1 previous similar message [ 7246.303094] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7246.355159] Lustre: server umount lustre-OST0000 complete [ 7249.376879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7249.377333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7249.381495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7249.387486] Lustre: Skipped 2 previous similar messages [ 7249.390760] Lustre: Skipped 1 previous similar message [ 7253.786205] Lustre: server umount lustre-MDT0000 complete [ 7254.495866] LustreError: 478527:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7254.500469] LustreError: 478527:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 7255.107367] LustreError: 479243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778846206 with bad export cookie 18357351840456772530 [ 7255.113170] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7255.116471] LustreError: 479243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7255.246740] Lustre: server umount lustre-MDT0001 complete [ 7258.660522] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7259.823061] Key type lgssc unregistered [ 7259.970565] LNet: 485944:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7259.975365] LNetError: 485944:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7259.987516] LNet: Removed LNI 192.168.204.126@tcp [ 7260.360450] Key type .llcrypt unregistered [ 7260.362080] Key type ._llcrypt unregistered [ 7268.050739] Key type ._llcrypt registered [ 7268.052995] Key type .llcrypt registered [ 7268.096941] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7268.589422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7268.638977] alg: No test for adler32 (adler32-zlib) [ 7269.498348] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 7269.590159] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 7271.183252] Key type lgssc registered [ 7271.645661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7284.815196] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 7287.786240] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7299.335773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7299.342087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7300.437133] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7300.449655] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7300.495050] Lustre: lustre-MDT0000: new disk, initializing [ 7300.522068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7300.529671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7301.787677] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7305.976940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7306.038894] Lustre: 490329:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7306.076820] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7306.081034] Lustre: Skipped 1 previous similar message [ 7306.127271] Lustre: lustre-MDT0001: new disk, initializing [ 7306.171344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7306.187657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7306.193368] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7307.540560] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7310.962053] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7312.077468] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7314.578602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7314.674648] Lustre: lustre-OST0000: new disk, initializing [ 7314.676254] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7314.678946] Lustre: 492070:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7314.709688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7315.331707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7315.340503] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7315.383778] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7316.972779] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7320.263765] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7321.656962] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7321.754717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7323.191579] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7323.267795] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7325.668535] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7325.669825] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7325.670762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7325.674139] LustreError: Skipped 1 previous similar message [ 7325.683304] Lustre: Skipped 1 previous similar message [ 7330.001804] Lustre: server umount lustre-OST0000 complete [ 7331.807862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7331.810185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7331.814208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7331.816702] Lustre: Skipped 1 previous similar message [ 7335.904739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7335.905958] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7337.363402] Lustre: server umount lustre-MDT0000 complete [ 7338.643321] LustreError: 491106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778846290 with bad export cookie 5350347945473982097 [ 7338.646901] LustreError: 491106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7338.647126] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7338.770837] Lustre: server umount lustre-MDT0001 complete [ 7346.488644] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 7349.648188] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7359.750520] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7207 sec ========== 07:58:30 (1778846310) [ 7360.327755] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 07:58:30 (1778846310) === [ 7361.452955] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 07:58:31 (1778846311) === [ 7374.155114] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7375.135691] Key type lgssc unregistered [ 7375.263415] LNet: 498504:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7375.265761] LNetError: 498504:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7375.273287] LNet: Removed LNI 192.168.204.126@tcp [ 7375.580821] Key type .llcrypt unregistered [ 7375.582355] Key type ._llcrypt unregistered