[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 566774238 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003162] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.007730] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.010142] LSM: Security Framework initializing [ 0.011052] Yama: becoming mindful. [ 0.012036] SELinux: Initializing. [ 0.013089] *** VALIDATE selinux *** [ 0.021637] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026239] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027145] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029113] *** VALIDATE tmpfs *** [ 0.031218] *** VALIDATE proc *** [ 0.032230] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.034290] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036149] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038046] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.042146] debug: unmapping init [mem 0xffffffff87259000-0xffffffff87260fff] [ 0.045000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045712] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046024] ... version: 2 [ 0.047012] ... bit width: 48 [ 0.048011] ... generic registers: 4 [ 0.049011] ... value mask: 0000ffffffffffff [ 0.050013] ... max period: 00007fffffffffff [ 0.051015] ... fixed-purpose events: 3 [ 0.052012] ... event mask: 000000070000000f [ 0.053321] rcu: Hierarchical SRCU implementation. [ 0.055452] smp: Bringing up secondary CPUs ... [ 0.056586] x86: Booting SMP configuration: [ 0.057024] .... node #0, CPUs: #1 #2 #3 [ 0.061083] smp: Brought up 1 node, 4 CPUs [ 0.063013] smpboot: Max logical packages: 1 [ 0.064015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.115494] node 0 deferred pages initialised in 49ms [ 0.119100] devtmpfs: initialized [ 0.120242] x86/mm: Memory block size: 128MB [ 0.126115] gcov: version magic: 0x41383552 [ 0.129292] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.132103] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.135304] pinctrl core: initialized pinctrl subsystem [ 0.137186] [ 0.137768] ************************************************************* [ 0.140011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.143011] ** ** [ 0.145011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.147012] ** ** [ 0.150013] ** This means that this kernel is built to expose internal ** [ 0.152012] ** IOMMU data structures, which may compromise security on ** [ 0.154011] ** your system. ** [ 0.157013] ** ** [ 0.159011] ** If you see this message and you are not debugging the ** [ 0.161012] ** kernel, report this immediately to your vendor! ** [ 0.164011] ** ** [ 0.166057] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168012] ************************************************************* [ 0.171998] NET: Registered protocol family 16 [ 0.174429] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.176084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.179084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.182094] cpuidle: using governor menu [ 0.183654] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.184780] PCI: Using configuration type 1 for base access [ 0.185139] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.193054] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.195027] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.199081] cryptd: max_cpu_qlen set to 1000 [ 0.202246] ACPI: Added _OSI(Module Device) [ 0.204016] ACPI: Added _OSI(Processor Device) [ 0.205010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.207013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.212019] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.218028] ACPI: Interpreter enabled [ 0.219050] ACPI: PM: (supports S0 S3 S4 S5) [ 0.221011] ACPI: Using IOAPIC for interrupt routing [ 0.223091] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.226403] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.236784] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.239037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.241019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.244084] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.249252] acpiphp: Slot [2] registered [ 0.251135] acpiphp: Slot [5] registered [ 0.252085] acpiphp: Slot [6] registered [ 0.254121] acpiphp: Slot [7] registered [ 0.256137] acpiphp: Slot [8] registered [ 0.258132] acpiphp: Slot [9] registered [ 0.260177] acpiphp: Slot [10] registered [ 0.262114] acpiphp: Slot [3] registered [ 0.264118] acpiphp: Slot [4] registered [ 0.266101] acpiphp: Slot [11] registered [ 0.268224] acpiphp: Slot [12] registered [ 0.270224] acpiphp: Slot [13] registered [ 0.272098] acpiphp: Slot [14] registered [ 0.274171] acpiphp: Slot [15] registered [ 0.276101] acpiphp: Slot [16] registered [ 0.277102] acpiphp: Slot [17] registered [ 0.279098] acpiphp: Slot [18] registered [ 0.281125] acpiphp: Slot [19] registered [ 0.283101] acpiphp: Slot [20] registered [ 0.285193] acpiphp: Slot [21] registered [ 0.286092] acpiphp: Slot [22] registered [ 0.288092] acpiphp: Slot [23] registered [ 0.290095] acpiphp: Slot [24] registered [ 0.292094] acpiphp: Slot [25] registered [ 0.293059] acpiphp: Slot [26] registered [ 0.294136] acpiphp: Slot [27] registered [ 0.296093] acpiphp: Slot [28] registered [ 0.298103] acpiphp: Slot [29] registered [ 0.300091] acpiphp: Slot [30] registered [ 0.302122] acpiphp: Slot [31] registered [ 0.303074] PCI host bridge to bus 0000:00 [ 0.305018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.308030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.311022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.314025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.317022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.320023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.322154] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.325273] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.328175] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.337763] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.343055] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.346017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.348016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.351030] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.354026] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.356697] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.360042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.362738] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.368014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.379930] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.383989] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.390427] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.407017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.428017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.477018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.492000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.512017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.526016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.566024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.590706] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.602027] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.611027] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.629028] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.638180] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.661019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.678016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.725018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.743113] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.754022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.766028] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.781027] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.793290] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.803017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.811018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.828016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.840222] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.843469] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.845726] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.849690] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.852566] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.857147] iommu: Default domain type: Passthrough [ 0.858000] SCSI subsystem initialized [ 0.859111] ACPI: bus type USB registered [ 0.860119] usbcore: registered new interface driver usbfs [ 0.862213] usbcore: registered new interface driver hub [ 0.864045] usbcore: registered new device driver usb [ 0.865178] pps_core: LinuxPPS API ver. 1 registered [ 0.867010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.870058] PTP clock support registered [ 0.872143] EDAC MC: Ver: 3.0.0 [ 0.874141] PCI: Using ACPI for IRQ routing [ 0.875986] NetLabel: Initializing [ 0.877011] NetLabel: domain hash size = 128 [ 0.879008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.880076] NetLabel: unlabeled traffic allowed by default [ 0.882139] vgaarb: loaded [ 0.884253] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.886013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.890115] clocksource: Switched to clocksource kvm-clock [ 0.997193] VFS: Disk quotas dquot_6.6.0 [ 0.998686] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.001027] *** VALIDATE ramfs *** [ 1.001889] *** VALIDATE hugetlbfs *** [ 1.003046] pnp: PnP ACPI init [ 1.005291] pnp: PnP ACPI: found 6 devices [ 1.025440] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.028381] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.030528] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.032656] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.034876] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.037210] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.039913] NET: Registered protocol family 2 [ 1.042252] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.046768] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.051383] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.057681] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.061109] TCP: Hash tables configured (established 65536 bind 65536) [ 1.063864] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.066996] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.069694] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.071878] NET: Registered protocol family 1 [ 1.074076] RPC: Registered named UNIX socket transport module. [ 1.075861] RPC: Registered udp transport module. [ 1.077379] RPC: Registered tcp transport module. [ 1.078855] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.081385] NET: Registered protocol family 44 [ 1.082959] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.084873] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.086906] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.089168] PCI: CLS 0 bytes, default 64 [ 1.091574] Unpacking initramfs... [ 2.552906] debug: unmapping init [mem 0xffff9bacbcc54000-0xffff9bacbffbffff] [ 2.556866] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.559093] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.561654] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.044845] Initialise system trusted keyrings [ 3.046553] Key type blacklist registered [ 3.048485] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.056867] zbud: loaded [ 3.059579] *** VALIDATE nfs *** [ 3.060752] *** VALIDATE nfs4 *** [ 3.062510] pstore: using deflate compression [ 3.065690] Platform Keyring initialized [ 3.162629] NET: Registered protocol family 38 [ 3.166619] Key type asymmetric registered [ 3.168700] Asymmetric key parser 'x509' registered [ 3.170994] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.173878] io scheduler mq-deadline registered [ 3.175610] io scheduler kyber registered [ 3.177445] io scheduler bfq registered [ 3.179379] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.182635] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.185848] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.188825] ACPI: Power Button [PWRF] [ 3.194468] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.201723] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.219720] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.231417] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.254753] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.286273] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.318409] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.323153] Non-volatile memory driver v1.3 [ 3.324926] Linux agpgart interface v0.103 [ 3.353360] virtio_blk virtio1: [vda] 150688 512-byte logical blocks (77.2 MB/73.6 MiB) [ 3.355625] vda: detected capacity change from 0 to 77152256 [ 3.369509] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.372444] vdb: detected capacity change from 0 to 1073741824 [ 3.396710] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.399666] vdc: detected capacity change from 0 to 2621440000 [ 3.434480] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.438674] vdd: detected capacity change from 0 to 2621440000 [ 3.454657] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.457812] vde: detected capacity change from 0 to 4294967296 [ 3.487163] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.490142] vdf: detected capacity change from 0 to 4294967296 [ 3.509779] libphy: Fixed MDIO Bus: probed [ 3.520511] usbcore: registered new interface driver usbserial_generic [ 3.523294] usbserial: USB Serial support registered for generic [ 3.527251] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.532397] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.534340] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.536817] mousedev: PS/2 mouse device common for all mice [ 3.539516] rtc_cmos 00:05: RTC can wake from S4 [ 3.544084] rtc_cmos 00:05: registered as rtc0 [ 3.544172] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.545702] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.552813] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.554426] intel_pstate: CPU model not supported [ 3.562309] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.567734] hid: raw HID events driver (C) Jiri Kosina [ 3.569764] usbcore: registered new interface driver usbhid [ 3.571593] usbhid: USB HID core driver [ 3.573514] drop_monitor: Initializing network drop monitor service [ 3.576100] Initializing XFRM netlink socket [ 3.578197] NET: Registered protocol family 10 [ 3.580804] Segment Routing with IPv6 [ 3.582369] NET: Registered protocol family 17 [ 3.585275] mpls_gso: MPLS GSO support [ 3.591594] RAS: Correctable Errors collector initialized. [ 3.593898] AVX version of gcm_enc/dec engaged. [ 3.595588] AES CTR mode by8 optimization enabled [ 3.698906] sched_clock: Marking stable (3698889141, 0)->(4632122332, -933233191) [ 3.702862] registered taskstats version 1 [ 3.705745] Loading compiled-in X.509 certificates [ 3.707675] zswap: loaded using pool lzo/zbud [ 3.728381] Key type big_key registered [ 3.739679] Key type encrypted registered [ 3.741291] ima: No TPM chip found, activating TPM-bypass! [ 3.743440] ima: Allocated hash algorithm: sha1 [ 3.745079] ima: No architecture policies found [ 3.746599] evm: Initialising EVM extended attributes: [ 3.748290] evm: security.selinux [ 3.749454] evm: security.ima [ 3.750438] evm: security.capability [ 3.751594] evm: HMAC attrs: 0x1 [ 3.753730] rtc_cmos 00:05: setting system clock to 2026-09-30 12:54:34 UTC (1790772874) [ 3.759764] debug: unmapping init [mem 0xffffffff88203000-0xffffffff883fffff] [ 3.762666] debug: unmapping init [mem 0xffffffff86f82000-0xffffffff87258fff] [ 3.770065] Write protecting the kernel read-only data: 28672k [ 3.773304] debug: unmapping init [mem 0xffffffff85603000-0xffffffff857fffff] [ 3.775957] debug: unmapping init [mem 0xffffffff85f14000-0xffffffff85ffffff] [ 3.809528] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.816979] systemd[1]: Detected virtualization kvm. [ 3.819159] systemd[1]: Detected architecture x86-64. [ 3.821335] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.846408] systemd[1]: No hostname configured. [ 3.848368] systemd[1]: Set hostname to . [ 3.850502] random: systemd: uninitialized urandom read (16 bytes read) [ 3.853135] systemd[1]: Initializing machine ID from random generator. [ 3.932062] random: ln: uninitialized urandom read (6 bytes read) [ 4.082502] random: systemd: uninitialized urandom read (16 bytes read) [ 4.085024] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.090213] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.095173] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.790185] device-mapper: uevent: version 1.0.3 [ 4.792912] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 5.545689] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.600812] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 5.995019] scsi host0: ata_piix [ 6.124306] scsi host1: ata_piix [ 6.125816] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.128232] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.261752] random: crng init done [ 10.264589] random: 7 urandom warning(s) missed due to ratelimiting [ 10.326656] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.033936] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.233054] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.473427] SELinux: Disabled at runtime. [ 12.534843] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.543951] systemd[1]: Detected virtualization kvm. [ 12.545708] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.002590] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.005667] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.008947] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.011769] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.014146] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.021519] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.027213] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ 13.150089] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.401267] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.774151] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.874543] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.983873] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.994167] EDAC sbridge: Ver: 1.1.2 [ 15.904221] Key type dns_resolver registered [ 16.215175] NFS: Registering the id_resolver key type [ 16.217361] Key type id_resolver registered [ 16.219071] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg245-server login: [ 51.097753] libcfs: loading out-of-tree module taints kernel. [ 51.168789] Key type ._llcrypt registered [ 51.170719] Key type .llcrypt registered [ 51.232536] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 71.969816] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 73.778154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 73.801889] alg: No test for adler32 (adler32-zlib) [ 75.515416] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 76.837136] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 78.720959] Key type lgssc registered [ 81.049229] Lustre: Echo OBD driver; http://www.lustre.org/ [ 85.380107] hrtimer: interrupt took 3097279 ns [ 100.314316] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 142.161319] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 157.506269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 157.546759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 159.011814] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 159.065345] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 159.218224] Lustre: lustre-MDT0000: new disk, initializing [ 159.355033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 159.381473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 165.049425] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 178.844672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 178.947872] Lustre: 6510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 178.991889] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 178.997151] Lustre: Skipped 1 previous similar message [ 179.067927] Lustre: lustre-MDT0001: new disk, initializing [ 179.146477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 179.168479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 179.186347] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 183.370423] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 188.047201] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 198.730207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 199.018433] Lustre: lustre-OST0000: new disk, initializing [ 199.026521] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 199.037328] Lustre: 8445:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 199.105576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 203.874475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 203.888865] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 203.983824] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 206.347458] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.148278] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 221.380197] Lustre: lustre-OST0001: new disk, initializing [ 221.397049] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 221.406925] Lustre: 9518:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 221.531926] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 228.144824] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 231.568098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 231.579640] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 231.724916] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 240.119805] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 252.841740] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 259.889615] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing check_logdir /tmp/testlogs/ [ 265.890477] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing yml_node [ 270.431799] Lustre: DEBUG MARKER: Client: 2.17.59.4 [ 273.309975] Lustre: DEBUG MARKER: MDS: 2.17.59.4 [ 276.302889] Lustre: DEBUG MARKER: OSS: 2.17.59.4 [ 277.933953] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 30 08:59:06 EDT 2026 [ 293.679229] Lustre: DEBUG MARKER: only running tests: 33-75 [ 295.101309] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 296.398738] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 312.296280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 312.306236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 312.320931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 313.826595] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 313.827084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 313.873762] Lustre: Skipped 3 previous similar messages [ 317.700436] Lustre: server umount lustre-MDT0000 complete [ 320.962167] LustreError: 9517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790773191 with bad export cookie 8861168875481570356 [ 320.965088] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 320.975247] LustreError: 9517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 324.079167] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 324.080183] LustreError: 9526:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 324.090704] Lustre: Skipped 2 previous similar messages [ 324.091606] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 324.125726] LustreError: 9526:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 325.089514] LustreError: 6515:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 325.101866] LustreError: 6515:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 330.211751] LustreError: 12205:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 330.211816] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 330.250134] LustreError: 12205:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 330.292214] Lustre: Skipped 4 previous similar messages [ 335.330213] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 335.886769] Lustre: server umount lustre-MDT0001 complete [ 340.815475] Lustre: server umount lustre-OST0000 complete [ 345.488849] Lustre: server umount lustre-OST0001 complete [ 355.711578] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 367.741391] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 408.902683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 409.161191] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 409.219325] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 409.381639] Lustre: lustre-MDT0000: new disk, initializing [ 409.514788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 409.541762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 415.233831] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 426.321483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 426.482550] Lustre: 15136:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 426.561834] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 426.564494] Lustre: Skipped 1 previous similar message [ 426.706570] Lustre: lustre-MDT0001: new disk, initializing [ 426.806599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 426.846518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 426.860187] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 431.641861] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 441.290112] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 447.451091] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 455.372878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 455.602865] Lustre: lustre-OST0000: new disk, initializing [ 455.605249] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 455.607508] Lustre: 17103:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 455.681871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 457.713561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 457.726553] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 457.849356] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 462.305173] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 471.457503] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 476.540832] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 476.786167] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 481.567229] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 481.778815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 483.812044] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 483.818381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 483.837720] Lustre: Skipped 1 previous similar message [ 483.846859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 483.851937] Lustre: Skipped 2 previous similar messages [ 488.428169] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 488.449592] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 489.775734] Lustre: server umount lustre-OST0000 complete [ 498.661542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 498.668025] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 498.673914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 498.674119] Lustre: Skipped 2 previous similar messages [ 498.678916] Lustre: Skipped 1 previous similar message [ 508.384761] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 508.635908] Lustre: server umount lustre-MDT0000 complete [ 508.906971] LustreError: 15147:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 508.931448] LustreError: 15147:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 512.247495] LustreError: 15128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790773382 with bad export cookie 8861168875481571882 [ 512.255476] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 512.261146] LustreError: 15128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 512.647641] Lustre: server umount lustre-MDT0001 complete [ 516.470951] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 09:03:05 (1790773385) [ 518.883774] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 520.297950] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 09:03:09 (1790773389) [ 527.591283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 528.098069] LustreError: 19515:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 528.175595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 532.980728] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 537.576954] LustreError: 19515:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 537.599553] LustreError: 19515:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 541.226340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 545.672443] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 556.517855] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 563.360102] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 570.672794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 578.738531] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 587.232943] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 597.985087] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 597.994883] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 598.003528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 598.012661] Lustre: Skipped 1 previous similar message [ 598.030896] Lustre: Skipped 4 previous similar messages [ 601.579911] Lustre: server umount lustre-OST0000 complete [ 608.228660] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 612.402404] Lustre: server umount lustre-MDT0000 complete [ 613.349522] LustreError: 19515:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 613.370276] LustreError: 19515:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 616.863049] LustreError: 19497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790773487 with bad export cookie 8861168875481572771 [ 616.876560] LustreError: 19497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 616.882337] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 617.325477] Lustre: server umount lustre-MDT0001 complete [ 629.850808] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 633.418835] Key type lgssc unregistered [ 633.700436] LNet: 23048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 633.710145] LNetError: 23048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 633.726545] LNet: Removed LNI 192.168.202.145@tcp [ 634.706555] Key type .llcrypt unregistered [ 634.711449] Key type ._llcrypt unregistered [ 657.015355] Key type ._llcrypt registered [ 657.019663] Key type .llcrypt registered [ 657.127242] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 657.991710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 658.004248] alg: No test for adler32 (adler32-zlib) [ 659.159964] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 659.364897] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 661.009769] Key type lgssc registered [ 662.049583] Lustre: Echo OBD driver; http://www.lustre.org/ [ 670.788589] loop: module loaded [ 670.898044] loop0: detected capacity change from 0 to 204800000 [ 671.148424] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 671.174628] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 671.425856] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 677.112564] loop0: detected capacity change from 0 to 204800000 [ 677.277773] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 677.297990] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 677.319517] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 682.219734] loop0: detected capacity change from 0 to 204800000 [ 687.317497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 687.353995] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 688.577836] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 688.642667] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 688.712239] Lustre: lustre-MDT0000: new disk, initializing [ 688.794124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 688.808557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 693.842861] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 702.218613] loop1: detected capacity change from 0 to 204800000 [ 708.030201] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 708.443155] Lustre: lustre-OST07c6: new disk, initializing [ 708.456538] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 708.467683] Lustre: Skipped 1 previous similar message [ 708.474786] Lustre: 26259:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 708.597541] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 714.863533] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 716.969085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 716.987424] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 717.027619] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 720.685641] Lustre: 25249:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 720.695591] Lustre: 25249:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 720.709550] Lustre: 25249:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 720.718173] Lustre: 25249:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 720.729121] Lustre: 25249:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 720.736367] Lustre: 25249:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 735.237263] Lustre: 25250:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 735.250392] Lustre: 25250:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 735.257580] Lustre: 25250:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 735.263500] Lustre: 25250:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 735.269115] Lustre: 25250:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 735.273923] Lustre: 25250:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 740.221363] Lustre: 25250:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 740.229547] Lustre: 25250:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 740.235201] Lustre: 25250:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 740.240936] Lustre: 25250:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 740.246286] Lustre: 25250:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 740.251300] Lustre: 25250:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message [ 740.256734] Lustre: 25250:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 740.260524] Lustre: 25250:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 740.270228] Lustre: 25250:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 740.276257] Lustre: 25250:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 740.282884] Lustre: 25250:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 740.291285] Lustre: 25250:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message [ 743.474076] Lustre: 25248:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 743.498117] Lustre: 25248:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 743.513136] Lustre: 25248:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 743.519073] Lustre: 25248:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 743.525043] Lustre: 25248:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 743.533553] Lustre: 25248:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message [ 743.538127] Lustre: 25248:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 743.548565] Lustre: 25248:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 743.556443] Lustre: 25248:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 743.562683] Lustre: 25248:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 743.572324] Lustre: 25248:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 743.578854] Lustre: 25248:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message [ 748.767587] Lustre: 25249:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 748.784555] Lustre: 25249:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 748.800674] Lustre: 25249:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 748.813096] Lustre: 25249:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 748.826632] Lustre: 25249:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 748.835805] Lustre: 25249:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message [ 748.843889] Lustre: 25249:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 748.851588] Lustre: 25249:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 748.858711] Lustre: 25249:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 748.864207] Lustre: 25249:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 748.872108] Lustre: 25249:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 748.879303] Lustre: 25249:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message [ 759.588799] Lustre: 25248:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 759.608608] Lustre: 25248:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 5 previous similar messages [ 759.625383] Lustre: 25248:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 759.632383] Lustre: 25248:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 759.642207] Lustre: 25248:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 759.648654] Lustre: 25248:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 759.656432] Lustre: 25248:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 759.665783] Lustre: 25248:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 759.682605] Lustre: 25248:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 759.691230] Lustre: 25248:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 759.701540] Lustre: 25248:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 759.716265] Lustre: 25248:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 808.930521] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 808.950420] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 814.051934] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 814.665570] Lustre: server umount lustre-OST07c6 complete [ 826.182448] Lustre: server umount lustre-MDT0000 complete [ 845.168258] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 848.037109] Key type lgssc unregistered [ 848.372388] LNet: 28818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 848.383632] LNetError: 28818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 848.406548] LNet: Removed LNI 192.168.202.145@tcp [ 849.364626] Key type .llcrypt unregistered [ 849.366939] Key type ._llcrypt unregistered [ 869.044585] Key type ._llcrypt registered [ 869.049184] Key type .llcrypt registered [ 869.307878] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 871.444253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 871.560621] alg: No test for adler32 (adler32-zlib) [ 872.956570] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 873.413565] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 875.168560] Key type lgssc registered [ 876.307211] Lustre: Echo OBD driver; http://www.lustre.org/ [ 885.360969] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 885.401562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 886.983651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 892.048686] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 896.484277] LustreError: 30104:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 896.525044] LustreError: 30104:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 901.601495] LustreError: 30103:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 901.780744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 902.297027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 907.497524] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 915.831937] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 922.526336] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 930.541725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 931.136472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 932.160854] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 938.739368] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 947.749589] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 967.141165] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 967.155912] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 967.162206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 970.929121] Lustre: server umount lustre-OST0000 complete [ 977.380629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 977.398011] Lustre: Skipped 2 previous similar messages [ 977.398563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 977.422236] Lustre: Skipped 1 previous similar message [ 982.136995] Lustre: server umount lustre-MDT0000 complete [ 982.498468] LustreError: 30098:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 982.506530] LustreError: 30098:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 986.436517] LustreError: 30085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790773857 with bad export cookie 12014922162693796476 [ 986.446419] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 986.449427] LustreError: 30085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 986.881980] Lustre: server umount lustre-MDT0001 complete [ 999.631561] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 1002.382743] Key type lgssc unregistered [ 1002.723759] LNet: 33842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.727358] LNetError: 33842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1003.750496] LNet: Removed LNI 192.168.202.145@tcp [ 1004.979521] Key type .llcrypt unregistered [ 1004.985024] Key type ._llcrypt unregistered [ 1023.617892] Key type ._llcrypt registered [ 1023.626671] Key type .llcrypt registered [ 1023.720580] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 1024.864332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1024.884593] alg: No test for adler32 (adler32-zlib) [ 1025.861334] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 1026.086573] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 1027.738328] Key type lgssc registered [ 1028.791805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1038.095721] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1038.123203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1039.685519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1044.785569] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1050.085951] LustreError: 35131:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1050.109435] LustreError: 35131:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1054.928107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1055.217914] LustreError: 35130:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1055.417655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1060.104710] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1067.066292] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1072.141952] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1077.897473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1078.203246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1083.709191] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1084.333598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1093.496218] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1104.868265] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1104.869485] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1104.875070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1104.877268] Lustre: Skipped 1 previous similar message [ 1109.996742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1110.009145] Lustre: Skipped 2 previous similar messages [ 1111.120828] Lustre: server umount lustre-OST0000 complete [ 1117.154289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1117.163248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1117.175823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1120.225921] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1120.231943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1120.248154] Lustre: Skipped 1 previous similar message [ 1121.659235] Lustre: server umount lustre-MDT0000 complete [ 1125.348319] LustreError: 35131:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1125.366461] LustreError: 35131:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1125.707356] LustreError: 35110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790773996 with bad export cookie 5069447952953213071 [ 1125.716446] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1125.723065] LustreError: 35110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1126.031597] Lustre: server umount lustre-MDT0001 complete [ 1137.950230] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 1139.885171] Key type lgssc unregistered [ 1140.135831] LNet: 38666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1140.150914] LNetError: 38666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1140.176875] LNet: Removed LNI 192.168.202.145@tcp [ 1141.147081] Key type .llcrypt unregistered [ 1141.148642] Key type ._llcrypt unregistered [ 1163.456188] Key type ._llcrypt registered [ 1163.460828] Key type .llcrypt registered [ 1163.601704] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 1164.571656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1164.601231] alg: No test for adler32 (adler32-zlib) [ 1165.639729] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 1165.790304] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 1167.416441] Key type lgssc registered [ 1168.259677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1177.684646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1177.705127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1179.193859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1183.927856] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1188.329885] LustreError: 40098:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1188.352911] LustreError: 40098:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1193.293465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1193.446393] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1193.674273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1199.021399] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1206.937963] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1213.649416] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1220.409532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1220.792447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1223.863555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1226.903885] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1234.943980] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1248.071170] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.45@tcp (stopping) [ 1249.768287] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1249.771122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1249.774400] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1249.785564] Lustre: Skipped 2 previous similar messages [ 1249.803588] Lustre: Skipped 2 previous similar messages [ 1251.929466] Lustre: server umount lustre-MDT0000 complete [ 1253.163336] LustreError: 42395:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1254.887183] LustreError: 40098:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1254.921960] LustreError: 40098:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1256.268275] LustreError: 40079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790774127 with bad export cookie 4721117012936520939 [ 1256.282684] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1258.272946] LustreError: 42395:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1258.314733] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.45@tcp (stopping) [ 1260.002394] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1262.715642] Lustre: server umount lustre-MDT0001 complete [ 1279.598625] Lustre: server umount lustre-OST0000 complete [ 1303.245244] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 1306.576168] Key type lgssc unregistered [ 1306.919569] LNet: 43958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1306.929459] LNetError: 43958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1306.962668] LNet: Removed LNI 192.168.202.145@tcp [ 1308.255354] Key type .llcrypt unregistered [ 1308.260730] Key type ._llcrypt unregistered [ 1334.185522] Key type ._llcrypt registered [ 1334.189366] Key type .llcrypt registered [ 1334.294197] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 1335.529121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1335.553686] alg: No test for adler32 (adler32-zlib) [ 1336.750960] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 1337.071990] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 1338.712248] Key type lgssc registered [ 1340.151979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1348.813832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1348.845283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1350.621985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1355.907377] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1360.878791] LustreError: 45395:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1360.906222] LustreError: 45395:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1365.689221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1366.069669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1371.570534] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1381.748783] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1389.426280] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1396.802752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1397.253372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1402.844656] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1410.830259] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1420.815629] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1431.020738] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1431.032261] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1431.033480] LustreError: Skipped 1 previous similar message [ 1431.043358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1431.079204] Lustre: Skipped 1 previous similar message [ 1431.882327] Lustre: lustre-OST0000: Not available for connect from 192.168.202.45@tcp (stopping) [ 1431.899633] Lustre: Skipped 1 previous similar message [ 1433.781586] Lustre: server umount lustre-OST0000 complete [ 1453.026490] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1453.041347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1453.067227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1455.587693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1455.598525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1459.100942] Lustre: server umount lustre-MDT0000 complete [ 1460.708816] LustreError: 45396:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1460.766730] LustreError: 45396:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1463.371733] LustreError: 45376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790774334 with bad export cookie 7765528836768483228 [ 1463.382414] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1463.390048] LustreError: 45376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1469.795826] Lustre: server umount lustre-MDT0001 complete [ 1481.948723] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 1485.085345] Key type lgssc unregistered [ 1485.424809] LNet: 49123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1485.436296] LNetError: 49123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1485.454754] LNet: Removed LNI 192.168.202.145@tcp [ 1486.583333] Key type .llcrypt unregistered [ 1486.585316] Key type ._llcrypt unregistered [ 1515.337466] Key type ._llcrypt registered [ 1515.340710] Key type .llcrypt registered [ 1515.531285] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 1516.754780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1516.781230] alg: No test for adler32 (adler32-zlib) [ 1518.368923] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 1518.682499] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 1520.384968] Key type lgssc registered [ 1521.870974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1532.546975] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1532.583713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1534.223953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1540.541098] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1543.667736] LustreError: 50572:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1543.692379] LustreError: 50572:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1548.768969] LustreError: 50571:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1550.021390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1550.366369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1556.091292] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1564.163032] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1569.806467] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1576.547115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1576.862893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1578.933332] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1583.869202] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1591.235501] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1601.125858] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1604.431699] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1617.206803] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-30 9h21m24s [ 1618.986432] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1621.991370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1621.997869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1622.016958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1623.079549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1623.083226] Lustre: Skipped 2 previous similar messages [ 1624.909923] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.45@tcp (stopping) [ 1627.452547] Lustre: server umount lustre-MDT0000 complete [ 1629.169881] LustreError: 50571:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1629.193966] LustreError: 50571:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1631.487557] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1632.031927] LustreError: 53258:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1638.178628] LustreError: 50568:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1638.190634] LustreError: 50568:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1638.945737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1639.181938] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1639.588587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1639.663179] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1640.630903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1645.234376] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1649.427831] Lustre: DEBUG MARKER: Wait for df (36969) ... [ 1651.028909] Lustre: DEBUG MARKER: done [ 1656.317591] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1656.332462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1656.337143] Lustre: Skipped 3 previous similar messages [ 1656.353135] Lustre: Skipped 2 previous similar messages [ 1661.414267] LustreError: 52393:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1661.521649] Lustre: server umount lustre-OST0000 complete [ 1666.531104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1666.540374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1666.552888] Lustre: Skipped 1 previous similar message [ 1666.573045] Lustre: Skipped 1 previous similar message [ 1672.145989] Lustre: server umount lustre-MDT0000 complete [ 1676.860912] LustreError: 53197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790774547 with bad export cookie 16769437464629721141 [ 1676.868438] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1676.878122] LustreError: 53197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1677.300784] Lustre: server umount lustre-MDT0001 complete [ 1688.939542] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 1692.761982] Key type lgssc unregistered [ 1693.189239] LNet: 55504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1693.193475] LNetError: 55504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1693.222713] LNet: Removed LNI 192.168.202.145@tcp [ 1694.226434] Key type .llcrypt unregistered [ 1694.232498] Key type ._llcrypt unregistered [ 1743.940688] Key type ._llcrypt registered [ 1743.944482] Key type .llcrypt registered [ 1744.086407] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 1745.528779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1745.583744] alg: No test for adler32 (adler32-zlib) [ 1746.585372] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 1746.757897] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 1748.536266] Key type lgssc registered [ 1750.161172] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1759.054247] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1759.105807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1760.500922] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1760.522689] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1760.683619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1765.447185] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1774.855307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1775.034194] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1775.119420] Lustre: 57727:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1775.479814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1781.441907] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1789.637458] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1795.279475] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1802.296666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1802.620858] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1802.822081] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1809.647520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1810.180362] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1819.252995] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1827.231752] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-30 9h24m55s [ 1828.743835] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1837.215972] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1839.906748] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1839.995180] LustreError: 59275:0:(mdt_handler.c:2345:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1860.436951] Lustre: lustre-MDT0000: Client 6697fe67-e872-4814-8933-26525dd9f780 (at 192.168.202.45@tcp) reconnecting [ 1880.024247] LustreError: 59275:0:(mdt_handler.c:2345:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1883.116952] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1886.410627] Lustre: DEBUG MARKER: done [ 1896.932623] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1896.938085] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1896.951812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1896.960246] Lustre: Skipped 1 previous similar message [ 1899.620019] Lustre: server umount lustre-OST0000 complete [ 1907.172575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1907.195124] Lustre: Skipped 1 previous similar message [ 1907.202231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1907.212307] Lustre: Skipped 1 previous similar message [ 1908.267156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1908.279922] Lustre: Skipped 1 previous similar message [ 1911.027046] Lustre: server umount lustre-MDT0000 complete [ 1915.296021] LustreError: 57720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790774786 with bad export cookie 12747452101149031421 [ 1915.303539] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1915.313042] LustreError: 57720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1915.733823] Lustre: server umount lustre-MDT0001 complete [ 1928.011288] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 1931.208642] Key type lgssc unregistered [ 1931.577105] LNet: 62066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1931.592052] LNetError: 62066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1932.648219] LNet: Removed LNI 192.168.202.145@tcp [ 1933.513887] Key type .llcrypt unregistered [ 1933.522728] Key type ._llcrypt unregistered [ 1980.773377] Key type ._llcrypt registered [ 1980.776330] Key type .llcrypt registered [ 1980.863634] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 1981.688098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1981.699981] alg: No test for adler32 (adler32-zlib) [ 1982.746690] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 1983.069954] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 1984.768285] Key type lgssc registered [ 1986.094056] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1990.580987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1990.609275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1991.922326] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1991.944532] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1992.103728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1993.102821] Lustre: Failing over lustre-MDT0000 [ 1993.308629] Lustre: server umount lustre-MDT0000 complete [ 2000.533474] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:27:49 (1790774869) [ 2007.110679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2007.426314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2012.741034] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2023.286944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2023.437228] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2023.488842] Lustre: 64888:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2023.888565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2030.576944] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2040.250379] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2047.242117] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2055.444710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2055.656873] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2056.033656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2061.520644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2063.847685] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2073.200755] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2079.863325] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2087.393728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2087.404414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2087.405988] Lustre: Skipped 2 previous similar messages [ 2087.417068] Lustre: Skipped 2 previous similar messages [ 2092.547480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2092.564169] Lustre: Skipped 2 previous similar messages [ 2097.634157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2097.647213] Lustre: Skipped 2 previous similar messages [ 2101.216308] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2101.489386] Lustre: server umount lustre-MDT0000 complete [ 2104.791146] LustreError: 64881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790774975 with bad export cookie 17659724086524051899 [ 2104.795258] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2104.798837] LustreError: 64881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2105.045749] Lustre: server umount lustre-MDT0001 complete [ 2108.486234] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2114.281578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2124.266623] Lustre: 63800:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790774978/real 1790774978] req@00000000b0c2cc28 x1877763532305024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790774994 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2124.294964] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2125.083438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2125.283652] Lustre: 63801:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790774980/real 1790774980] req@000000006b51f4a0 x1877763532305152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790774996 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2128.353253] Lustre: 63802:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790774983/real 1790774983] req@0000000036b9884d x1877763532305280/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790774999 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2129.945674] LustreError: 66738:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2129.976616] LustreError: 66738:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2130.168310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2130.347258] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2131.232426] Lustre: 63803:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790774985/real 1790774985] req@00000000307a161a x1877763532305536/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790775001 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2133.239849] LustreError: 66739:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2133.252989] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2133.277045] LustreError: 66739:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2134.308717] LustreError: 66738:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2134.327783] LustreError: 66738:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2135.265562] Lustre: 63801:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790774990/real 1790774990] req@000000008ceee9a9 x1877763532305664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790775006 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2136.201800] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2139.624611] LustreError: 66739:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2139.666982] LustreError: 66739:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2144.745219] LustreError: 68664:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2144.755382] LustreError: 68664:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2146.343684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2146.815809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2151.940611] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2152.264543] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2159.819782] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2166.324944] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2177.515222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2177.519102] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2177.524914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2177.541489] Lustre: Skipped 2 previous similar messages [ 2179.979956] Lustre: server umount lustre-MDT0000 complete [ 2182.642549] LustreError: 69309:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2182.669144] LustreError: 69309:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2184.214364] LustreError: 64881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790775054 with bad export cookie 17659724086524054174 [ 2184.239100] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2184.242406] LustreError: 64881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2184.659360] Lustre: server umount lustre-MDT0001 complete [ 2191.391487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2201.815355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2204.129040] Lustre: 63802:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790775058/real 1790775058] req@00000000547174c3 x1877763532333952/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790775074 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2204.162131] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2209.898288] LustreError: 69399:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2210.040682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2210.143074] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2215.629582] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2224.281400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2229.740043] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2229.936675] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2232.898682] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2237.763743] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2243.485908] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2250.216723] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2250.223585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2250.237204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2250.240100] Lustre: Skipped 2 previous similar messages [ 2256.424047] Lustre: server umount lustre-MDT0000 complete [ 2258.402785] LustreError: 71347:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2258.415784] LustreError: 71347:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 2260.762571] LustreError: 64882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790775131 with bad export cookie 17659724086524055553 [ 2260.775057] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2260.789074] LustreError: 64882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2261.150661] Lustre: server umount lustre-MDT0001 complete [ 2269.571957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2273.676240] Lustre: DEBUG MARKER: files compared the same [ 2286.124725] Lustre: server umount lustre-OST0000 complete [ 2305.373273] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 2309.623941] Key type lgssc unregistered [ 2309.983699] LNet: 74623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2309.990249] LNetError: 74623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2310.010935] LNet: Removed LNI 192.168.202.145@tcp [ 2311.254709] Key type .llcrypt unregistered [ 2311.257539] Key type ._llcrypt unregistered [ 2342.455745] Key type ._llcrypt registered [ 2342.458930] Key type .llcrypt registered [ 2342.577975] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 2343.860034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2343.913726] alg: No test for adler32 (adler32-zlib) [ 2345.181962] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 2345.618355] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 2347.312732] Key type lgssc registered [ 2348.896882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2357.470902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2357.508911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2359.115235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2364.467126] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug +malloc all [ 2368.487191] LustreError: 76058:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2368.512885] LustreError: 76058:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2373.600808] LustreError: 76059:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2374.391960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2375.029395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2380.466510] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug +malloc all [ 2389.054839] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2395.767641] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2403.093603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2403.553975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2404.583751] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2410.042408] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug +malloc all [ 2417.666592] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2430.437130] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2430.444479] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2430.453196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2435.564059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2435.576919] Lustre: Skipped 2 previous similar messages [ 2440.673268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2440.682313] Lustre: Skipped 1 previous similar message [ 2441.186334] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2441.365248] Lustre: server umount lustre-OST0000 complete [ 2446.821979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2446.831349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2446.846950] Lustre: Skipped 1 previous similar message [ 2446.854417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2450.915270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2450.920632] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2452.498882] Lustre: server umount lustre-MDT0000 complete [ 2456.043294] LustreError: 76058:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2456.076137] LustreError: 76058:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2456.993402] LustreError: 76038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790775327 with bad export cookie 10202448897011625377 [ 2457.012469] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2457.362307] Lustre: server umount lustre-MDT0001 complete [ 2469.517181] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 2472.989634] Key type lgssc unregistered [ 2473.342778] LNet: 79593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2473.348613] LNetError: 79593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2473.363367] LNet: Removed LNI 192.168.202.145@tcp [ 2474.263629] Key type .llcrypt unregistered [ 2474.265606] Key type ._llcrypt unregistered [ 2499.226499] Key type ._llcrypt registered [ 2499.229481] Key type .llcrypt registered [ 2499.421145] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 2500.731308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2501.030799] alg: No test for adler32 (adler32-zlib) [ 2501.977882] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 2502.125497] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 2503.768228] Key type lgssc registered [ 2505.356815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2515.054334] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2515.087155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2530.592439] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2535.712327] LustreError: 80999:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.145@tcp: failed processing log, type 4: rc = -110 [ 2562.400966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2570.322974] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2578.628359] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2589.157272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2589.507421] Lustre: *** cfs_fail_loc=706, val=0*** [ 2590.249567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2590.338540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2595.310523] LustreError: 81025:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2595.349842] LustreError: 81025:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2595.671080] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2596.338471] LustreError: 81024:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2596.361405] LustreError: 81024:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2601.466062] LustreError: 82084:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2601.514135] LustreError: 82084:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2606.566615] LustreError: 81025:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2606.593117] LustreError: 81025:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2606.676426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2607.063046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2612.783249] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2625.576104] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2635.206864] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2641.377921] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2641.392794] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2641.415149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2642.914988] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2642.918191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2642.922565] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2642.958592] Lustre: Skipped 1 previous similar message [ 2648.034897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2648.048341] Lustre: Skipped 1 previous similar message [ 2652.128152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2652.366358] Lustre: server umount lustre-OST0000 complete [ 2658.273471] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2658.277363] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2658.284604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2658.293994] Lustre: Skipped 1 previous similar message [ 2663.394360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2663.409624] Lustre: Skipped 2 previous similar messages [ 2671.072230] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2671.313818] Lustre: server umount lustre-MDT0000 complete [ 2672.611034] LustreError: 82089:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2672.623883] LustreError: 82089:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2674.915567] LustreError: 81005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790775545 with bad export cookie 14721471620834130816 [ 2674.922769] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2674.926286] LustreError: 81005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2675.449283] Lustre: server umount lustre-MDT0001 complete [ 2687.981269] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 2692.060422] Key type lgssc unregistered [ 2692.278820] LNet: 84624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2692.285672] LNetError: 84624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2692.302649] LNet: Removed LNI 192.168.202.145@tcp [ 2693.450093] Key type .llcrypt unregistered [ 2693.459175] Key type ._llcrypt unregistered [ 2722.946216] Key type ._llcrypt registered [ 2722.950348] Key type .llcrypt registered [ 2723.142451] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 2725.713339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2725.808612] alg: No test for adler32 (adler32-zlib) [ 2727.355445] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 2727.775824] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 2729.656892] Key type lgssc registered [ 2730.698294] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2740.795631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2740.813120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2746.021941] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2754.654749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2755.201472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2760.554342] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2760.677462] LustreError: 86745:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2760.702177] LustreError: 86745:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2764.771430] LustreError: 86743:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2764.782313] LustreError: 86743:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2769.893301] LustreError: 86745:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2769.916650] LustreError: 86745:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2769.944792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2770.452662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2775.524748] LustreError: 86748:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2775.537916] LustreError: 86748:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 2777.279257] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2779.619369] LustreError: 86749:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2779.658474] LustreError: 86749:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2786.977892] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2786.992282] Lustre: MGS: Client 72fbe1d9-59d0-4f91-999a-d15255fe06e6 (at 0@lo) reconnecting [ 2787.016238] LustreError: 86759:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2787.258804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2787.384369] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2792.440456] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2809.843605] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2809.858211] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2809.870375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2812.904237] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2812.911750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2812.917956] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2812.942709] Lustre: Skipped 1 previous similar message [ 2815.128774] Lustre: server umount lustre-OST0000 complete [ 2823.141837] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2823.145754] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2823.160290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2823.170802] Lustre: Skipped 1 previous similar message [ 2823.197567] Lustre: Skipped 1 previous similar message [ 2828.257288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2828.268278] Lustre: Skipped 1 previous similar message [ 2833.377179] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2833.378024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2833.392804] Lustre: Skipped 1 previous similar message [ 2833.484677] Lustre: server umount lustre-MDT0000 complete [ 2837.480322] LustreError: 86749:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2837.515132] LustreError: 86749:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2838.304353] Lustre: server umount lustre-MDT0001 complete [ 2843.047386] Lustre: server umount MGS complete [ 2855.117173] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:42:03 (1790775723) [ 2883.731933] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 2894.434529] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 2931.321428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2936.570444] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2945.834486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2946.103808] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2946.186647] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2946.358350] Lustre: lustre-MDT0001: new disk, initializing [ 2946.458538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2946.519086] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2946.530270] Lustre: 93937:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2946.559045] LustreError: 93937:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2951.486834] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2962.191484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2962.545659] Lustre: lustre-OST0000: new disk, initializing [ 2962.551305] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2962.559206] Lustre: 94704:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2962.621579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2968.335665] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2973.793756] Lustre: 95212:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000c7e53781 x1877764313491968/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1790775860 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2979.009115] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2986.547928] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2986.567457] Lustre: MGS: Client df278943-f26f-4c67-ab3f-dcc30e091bc4 (at 0@lo) reconnecting [ 2986.613301] Lustre: 93205:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2986.625503] LustreError: 93923:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2986.648871] LustreError: 93923:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2986.662706] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2986.750253] Lustre: lustre-MDT0000: new disk, initializing [ 2986.915892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2986.941221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2987.054331] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2988.019925] Lustre: 95772:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000a4e14ac1 x1877764313498624/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1790775874 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2992.031207] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2994.168063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2994.180231] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2997.104595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2997.115145] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2997.234277] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 3022.819327] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3022.834772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3022.847285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3028.777430] Lustre: server umount lustre-OST0000 complete [ 3035.112833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3035.120405] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3035.153850] Lustre: Skipped 2 previous similar messages [ 3038.838944] Lustre: server umount lustre-MDT0000 complete [ 3040.226028] LustreError: 93909:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3040.248965] LustreError: 93909:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3050.055104] Lustre: server umount lustre-MDT0001 complete [ 3054.386734] Lustre: server umount MGS complete [ 3067.566727] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:45:35 (1790775935) [ 3088.227250] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 3091.533959] Key type lgssc unregistered [ 3091.870490] LNet: 98270:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3091.896608] LNetError: 98270:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3091.918153] LNet: Removed LNI 192.168.202.145@tcp [ 3092.980843] Key type .llcrypt unregistered [ 3092.985891] Key type ._llcrypt unregistered [ 3111.159292] Key type ._llcrypt registered [ 3111.161292] Key type .llcrypt registered [ 3111.274646] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 3112.539418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3112.551610] alg: No test for adler32 (adler32-zlib) [ 3113.561053] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 3113.789818] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 3115.472220] Key type lgssc registered [ 3116.646510] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3124.780492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3124.823123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3126.276809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3130.832733] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3135.470614] LustreError: 99512:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3135.510713] LustreError: 99512:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3140.586927] LustreError: 99511:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3140.960909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3141.344857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3146.653141] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3153.358981] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3159.290613] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3165.579954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3165.876570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3170.992856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 3172.094663] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3180.443763] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3188.473929] Lustre: server umount lustre-OST0000 complete [ 3196.900183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3196.910677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3196.916633] Lustre: Skipped 1 previous similar message [ 3202.019181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3202.029799] Lustre: Skipped 1 previous similar message [ 3203.185670] Lustre: server umount lustre-MDT0000 complete [ 3207.147638] LustreError: 99506:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3207.163575] LustreError: 99506:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3207.710262] LustreError: 99493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790776078 with bad export cookie 14319320035989406038 [ 3207.714787] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3207.732361] LustreError: 99493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3217.998367] Lustre: server umount lustre-MDT0001 complete [ 3229.179917] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 3231.881392] Key type lgssc unregistered [ 3232.165734] LNet: 103043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3232.176133] LNetError: 103043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3232.191772] LNet: Removed LNI 192.168.202.145@tcp [ 3233.203677] Key type .llcrypt unregistered [ 3233.206202] Key type ._llcrypt unregistered [ 3248.515636] Key type ._llcrypt registered [ 3248.525301] Key type .llcrypt registered [ 3248.630159] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 3249.583731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3249.708059] alg: No test for adler32 (adler32-zlib) [ 3250.814478] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 3251.152717] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 3252.857944] Key type lgssc registered [ 3254.415639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3261.829789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3261.839503] LustreError: 104133:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3261.865629] LustreError: 104137:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3261.879584] LustreError: 104133:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4971 [ 3261.893375] LustreError: 104133:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3261.908091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3261.914386] LustreError: 104133:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3263.523851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3270.083495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3270.317224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3274.620372] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3279.682507] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing lsmod [ 3282.130188] LustreError: 105332:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3282.161553] LustreError: 105341:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3282.168717] LustreError: 105332:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4969 [ 3282.173697] LustreError: 105332:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3282.185949] LustreError: 105332:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3282.192829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3282.690607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3285.985622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3285.992292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3285.999366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3288.034338] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3288.039276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3288.062356] Lustre: Skipped 1 previous similar message [ 3288.070473] Lustre: Skipped 2 previous similar messages [ 3291.704068] Lustre: server umount lustre-MDT0000 complete [ 3293.163768] LustreError: 104173:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3293.194690] LustreError: 104173:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3295.633422] LustreError: 104150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790776166 with bad export cookie 16075777774338022653 [ 3295.634924] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3295.656418] LustreError: 104150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3296.122470] Lustre: server umount lustre-MDT0001 complete [ 3310.992096] Lustre: server umount lustre-OST0000 complete [ 3320.255735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3320.775065] LustreError: 106404:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3320.844550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3325.925505] LustreError: 106405:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3326.080804] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3331.043736] LustreError: 106404:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3335.027856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3335.348259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3340.383667] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3348.218332] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3354.433318] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3361.379487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3361.979242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3366.073688] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3369.485968] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3378.277814] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3386.852096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3386.870215] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3386.882416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3389.926097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3389.930029] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3389.935699] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3391.552370] Lustre: server umount lustre-OST0000 complete [ 3397.089855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3397.105701] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3397.122957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3397.125485] Lustre: Skipped 1 previous similar message [ 3401.608227] Lustre: server umount lustre-MDT0000 complete [ 3405.231719] LustreError: 106386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790776275 with bad export cookie 16075777774338023339 [ 3405.238340] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3405.245883] LustreError: 106386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3405.498910] Lustre: server umount lustre-MDT0001 complete [ 3415.224818] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 3417.951799] Key type lgssc unregistered [ 3418.273872] LNet: 109937:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3418.278453] LNetError: 109937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3418.294500] LNet: Removed LNI 192.168.202.145@tcp [ 3419.293267] Key type .llcrypt unregistered [ 3419.297984] Key type ._llcrypt unregistered [ 3446.239210] Key type ._llcrypt registered [ 3446.241392] Key type .llcrypt registered [ 3446.317290] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 3447.263657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3447.354897] alg: No test for adler32 (adler32-zlib) [ 3448.444876] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 3448.608303] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 3450.304742] Key type lgssc registered [ 3452.015318] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3461.154494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3461.167806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3462.428707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3467.559788] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3472.866454] LustreError: 111372:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3472.881806] LustreError: 111372:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3476.671179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3477.307125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3481.451130] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3488.843365] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3494.698995] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3501.573250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3502.053891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3508.231653] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 3509.429486] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3517.172847] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3522.762709] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3526.689454] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3532.774222] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3532.779167] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3532.787412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3533.800310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3533.800674] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3536.949650] Lustre: server umount lustre-OST0000 complete [ 3544.034168] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3544.039408] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3544.073097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3544.076911] Lustre: Skipped 1 previous similar message [ 3544.108325] Lustre: Skipped 2 previous similar messages [ 3549.156237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3549.164549] Lustre: Skipped 1 previous similar message [ 3554.275839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3554.285450] Lustre: Skipped 1 previous similar message [ 3557.344368] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3557.635208] Lustre: server umount lustre-MDT0000 complete [ 3558.378097] LustreError: 112612:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3558.419350] LustreError: 112612:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3561.834507] LustreError: 111351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790776432 with bad export cookie 12900027541884758208 [ 3561.847263] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3562.096679] Lustre: server umount lustre-MDT0001 complete [ 3573.093642] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 3575.913299] Key type lgssc unregistered [ 3576.335982] LNet: 115009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3576.342888] LNetError: 115009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3576.363267] LNet: Removed LNI 192.168.202.145@tcp [ 3577.149234] Key type .llcrypt unregistered [ 3577.151399] Key type ._llcrypt unregistered [ 3596.089648] Key type ._llcrypt registered [ 3596.092323] Key type .llcrypt registered [ 3596.207839] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 3597.379688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3597.400978] alg: No test for adler32 (adler32-zlib) [ 3598.717298] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 3598.996531] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 3600.720206] Key type lgssc registered [ 3602.564830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3610.492646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3610.516496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3611.864313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3616.367083] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3621.348812] LustreError: 116252:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3621.393448] LustreError: 116252:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3626.495493] LustreError: 116253:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3627.185466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3628.009577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3633.205875] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3642.015443] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3648.381722] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3657.298458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3657.716894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3665.070443] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3672.293135] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3682.288632] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3692.514458] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3692.523739] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3692.528435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3692.530676] Lustre: Skipped 1 previous similar message [ 3697.275756] Lustre: server umount lustre-OST0000 complete [ 3702.759270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3702.776462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3702.782127] Lustre: Skipped 2 previous similar messages [ 3707.881438] LustreError: 117381:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3707.929684] Lustre: server umount lustre-MDT0000 complete [ 3711.791436] LustreError: 116233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790776582 with bad export cookie 9469883684379654294 [ 3711.795887] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3711.804038] LustreError: 116233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3712.083278] Lustre: server umount lustre-MDT0001 complete [ 3723.984080] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 3726.657029] Key type lgssc unregistered [ 3726.972833] LNet: 119809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3726.985980] LNetError: 119809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3727.002917] LNet: Removed LNI 192.168.202.145@tcp [ 3727.845466] Key type .llcrypt unregistered [ 3727.847911] Key type ._llcrypt unregistered [ 3752.810960] Key type ._llcrypt registered [ 3752.814328] Key type .llcrypt registered [ 3752.963982] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 3754.745638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3754.772730] alg: No test for adler32 (adler32-zlib) [ 3755.866406] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 3756.161701] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 3757.864436] Key type lgssc registered [ 3759.077599] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3767.519191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3767.537756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3769.015864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3774.047861] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3778.534785] LustreError: 121243:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3778.550136] LustreError: 121243:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3781.606541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3782.093648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3786.533866] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3794.502314] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3801.948461] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3811.684608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3811.983685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3814.144987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3819.698333] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3829.279315] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3835.944779] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3840.785404] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3840.789995] Lustre: Skipped 2 previous similar messages [ 3845.229864] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3845.316848] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3846.268312] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3846.286184] Lustre: Skipped 2 previous similar messages [ 3853.880713] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3853.887633] Lustre: Skipped 1 previous similar message [ 3858.602909] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.45@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3858.609032] Lustre: Skipped 2 previous similar messages [ 3859.248314] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.45@tcp 192.168.0.[2,10]@tcp [ 3870.625840] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.45/24@tcp in log lustre-MDT0000 [ 3870.633189] Lustre: Skipped 2 previous similar messages [ 3880.032193] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.45/24@tcp [ 3880.039753] Lustre: Skipped 1 previous similar message [ 3886.055147] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3886.055340] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3886.066373] LustreError: Skipped 1 previous similar message [ 3886.101378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3888.001355] Lustre: server umount lustre-OST0000 complete [ 3893.225542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3893.262276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3897.645835] Lustre: server umount lustre-MDT0000 complete [ 3898.339980] LustreError: 122251:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3898.359995] LustreError: 122251:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3901.805336] LustreError: 121223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790776772 with bad export cookie 8156617534235895487 [ 3901.808771] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3901.813186] LustreError: 121223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3912.472457] Lustre: server umount lustre-MDT0001 complete [ 3925.479616] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 3929.023462] Key type lgssc unregistered [ 3929.273459] LNet: 126237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3929.282978] LNetError: 126237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3929.297987] LNet: Removed LNI 192.168.202.145@tcp [ 3930.691486] Key type .llcrypt unregistered [ 3930.694870] Key type ._llcrypt unregistered [ 3966.488683] Key type ._llcrypt registered [ 3966.496990] Key type .llcrypt registered [ 3966.636104] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 3967.973537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3968.333458] alg: No test for adler32 (adler32-zlib) [ 3969.444173] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 3969.722893] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 3971.440234] Key type lgssc registered [ 3972.939805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3981.681604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3981.691747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3983.339724] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3983.354175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3983.396411] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.45/24@tcp [ 3988.613408] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3993.580422] LustreError: 127815:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3993.601352] LustreError: 127815:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3998.691097] LustreError: 127814:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3998.837327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3999.243149] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3999.264392] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.45/24@tcp [ 3999.353976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4004.475223] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4011.627531] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4017.518562] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4026.025400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4026.628764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4028.765069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 4034.130699] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4042.498816] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4054.497812] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4054.506271] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4054.507902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4054.518818] Lustre: Skipped 1 previous similar message [ 4054.548753] Lustre: Skipped 1 previous similar message [ 4059.634357] LustreError: 129760:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4059.793260] Lustre: server umount lustre-OST0000 complete [ 4064.739138] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4064.748633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4064.757086] Lustre: Skipped 1 previous similar message [ 4064.773222] Lustre: Skipped 1 previous similar message [ 4069.863536] LustreError: 127815:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4069.900701] LustreError: 127815:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4069.931871] Lustre: server umount lustre-MDT0000 complete [ 4074.186519] LustreError: 127794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790776944 with bad export cookie 1559076138434392837 [ 4074.199922] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4074.401326] Lustre: server umount lustre-MDT0001 complete [ 4091.957217] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 4095.105985] Key type lgssc unregistered [ 4095.409496] LNet: 131432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4095.414387] LNetError: 131432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4095.433221] LNet: Removed LNI 192.168.202.145@tcp [ 4096.525556] Key type .llcrypt unregistered [ 4096.527861] Key type ._llcrypt unregistered [ 4125.649852] Key type ._llcrypt registered [ 4125.652971] Key type .llcrypt registered [ 4125.822939] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 4144.898580] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 4146.331650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4146.561163] alg: No test for adler32 (adler32-zlib) [ 4147.713932] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 4147.939228] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 4149.712458] Key type lgssc registered [ 4151.330281] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4189.941218] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4190.003221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4191.411685] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4191.449877] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4191.578802] Lustre: lustre-MDT0000: new disk, initializing [ 4191.759658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4191.801309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4197.241227] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4208.271882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4208.389966] Lustre: 134852:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4208.436234] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4208.441797] Lustre: Skipped 1 previous similar message [ 4208.488782] Lustre: lustre-MDT0001: new disk, initializing [ 4208.590641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4208.666796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4208.689516] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4214.482681] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4224.889543] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4230.948347] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4239.830401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4240.028668] Lustre: lustre-OST0000: new disk, initializing [ 4240.032710] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4240.038814] Lustre: 136824:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4240.114398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4246.106928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4246.125381] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4246.287965] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4247.968692] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4257.796294] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4262.858603] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4263.017408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4267.238348] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4267.388685] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4272.097854] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4272.108620] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4272.119033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4272.148302] Lustre: Skipped 1 previous similar message [ 4277.221983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4277.238304] Lustre: Skipped 2 previous similar messages [ 4282.342562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4282.352895] Lustre: Skipped 1 previous similar message [ 4283.361725] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4283.519718] Lustre: server umount lustre-OST0000 complete [ 4287.457948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4287.477332] Lustre: Skipped 1 previous similar message [ 4287.477533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4287.502915] Lustre: Skipped 1 previous similar message [ 4292.577874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4292.587908] Lustre: Skipped 1 previous similar message [ 4293.511904] Lustre: server umount lustre-MDT0000 complete [ 4297.455367] LustreError: 136823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790777168 with bad export cookie 14301092605514095168 [ 4297.456073] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4297.468293] LustreError: 136823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4297.981451] Lustre: server umount lustre-MDT0001 complete [ 4306.214932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4306.602400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4311.708713] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4312.045540] LustreError: 139040:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4312.057783] LustreError: 139040:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4317.153087] LustreError: 139041:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4320.965132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4321.341891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4326.063282] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4334.891602] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4341.527247] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4349.967753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4350.392474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4357.722180] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4366.598090] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4371.958364] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4372.224461] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4377.596264] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4377.936956] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4387.462379] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4387.628233] Lustre: lustre-OST0001: new disk, initializing [ 4387.631746] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4387.637802] Lustre: 142417:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4387.753701] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4394.081335] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4395.572419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4395.586345] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4395.620803] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4401.891105] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4402.230576] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4406.875460] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4407.103469] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4415.093059] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff89d243569000.ost_server_uuid 50 [ 4417.490514] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff89d243569000.ost_server_uuid in IDLE state after 0 sec [ 4431.337853] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4431.339255] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4431.356349] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4431.371597] Lustre: Skipped 1 previous similar message [ 4431.384903] Lustre: Skipped 1 previous similar message [ 4433.569175] Lustre: server umount lustre-OST0001 complete [ 4436.472948] LustreError: 140936:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4436.483336] LustreError: 140936:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4438.500107] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4438.509186] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4441.594098] LustreError: 140862:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4444.455898] Lustre: server umount lustre-OST0000 complete [ 4449.249396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4449.259465] LustreError: Skipped 1 previous similar message [ 4449.266098] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4449.282016] Lustre: Skipped 1 previous similar message [ 4449.287978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4449.293112] Lustre: Skipped 4 previous similar messages [ 4454.490372] Lustre: server umount lustre-MDT0000 complete [ 4456.931235] LustreError: 139035:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4456.939339] LustreError: 139035:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4458.347143] LustreError: 139022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790777329 with bad export cookie 14301092605514096050 [ 4458.352265] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4458.359628] LustreError: 139022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4458.675943] Lustre: server umount lustre-MDT0001 complete [ 4480.133969] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 4483.304377] Key type lgssc unregistered [ 4483.694555] LNet: 145417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4483.699337] LNetError: 145417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4483.738455] LNet: Removed LNI 192.168.202.145@tcp [ 4484.828794] Key type .llcrypt unregistered [ 4484.835284] Key type ._llcrypt unregistered [ 4528.196176] Key type ._llcrypt registered [ 4528.198456] Key type .llcrypt registered [ 4528.311223] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 4545.427574] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 4547.602588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4547.707164] alg: No test for adler32 (adler32-zlib) [ 4548.935806] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 4549.231323] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 4550.960303] Key type lgssc registered [ 4552.236944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4588.518824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4588.554106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4589.894194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4589.933342] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4590.023060] Lustre: lustre-MDT0000: new disk, initializing [ 4590.108636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4590.132779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4595.622211] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4607.390806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4607.549879] Lustre: 149591:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4607.603945] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4607.607948] Lustre: Skipped 1 previous similar message [ 4607.661766] Lustre: lustre-MDT0001: new disk, initializing [ 4607.767425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4607.821955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4607.832217] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4613.010442] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4623.322804] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4629.786549] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4637.126453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4637.420643] Lustre: lustre-OST0000: new disk, initializing [ 4637.423623] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4637.435550] Lustre: 151561:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4637.522048] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4643.406463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4643.421401] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4643.557310] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4645.290987] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4658.252522] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4663.728392] Lustre: Failing over lustre-OST0000 [ 4664.289270] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4664.292676] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4664.309097] LustreError: Skipped 1 previous similar message [ 4664.310759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4664.370088] Lustre: Skipped 1 previous similar message [ 4665.683374] Lustre: lustre-OST0000: Not available for connect from 192.168.202.45@tcp (stopping) [ 4665.691464] Lustre: Skipped 1 previous similar message [ 4666.054486] Lustre: server umount lustre-OST0000 complete [ 4685.888371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4686.214170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4686.229379] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4693.307724] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4697.184607] Lustre: Failing over lustre-MDT0000 [ 4697.218896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4697.229644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4697.247274] Lustre: Skipped 2 previous similar messages [ 4697.274284] Lustre: Skipped 2 previous similar messages [ 4697.728401] Lustre: server umount lustre-MDT0000 complete [ 4704.742215] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4705.567318] LustreError: 149600:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4705.588031] LustreError: 149600:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 4706.602685] LustreError: 149599:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4709.861246] LustreError: 149604:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4709.893574] LustreError: 149604:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4714.977600] LustreError: 149603:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4715.021408] LustreError: 149603:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4718.376083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4718.573992] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4718.884391] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4718.951558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4719.018856] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4721.952908] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4724.222162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4724.244291] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4724.316884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4724.435881] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4734.436045] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4734.455367] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4734.457495] LustreError: Skipped 1 previous similar message [ 4734.460908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4734.505186] Lustre: Skipped 1 previous similar message [ 4739.556898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4739.564199] Lustre: Skipped 2 previous similar messages [ 4739.757138] Lustre: server umount lustre-OST0000 complete [ 4744.676210] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4744.692919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4744.703422] Lustre: Skipped 1 previous similar message [ 4749.798218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4749.809037] Lustre: Skipped 3 previous similar messages [ 4750.548222] Lustre: server umount lustre-MDT0000 complete [ 4754.917219] LustreError: 149598:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4754.937384] LustreError: 149598:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4755.115741] LustreError: 149584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790777625 with bad export cookie 11272807393678520342 [ 4755.120256] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4755.134329] LustreError: 149584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4755.721974] Lustre: server umount lustre-MDT0001 complete [ 4769.169777] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 4772.238189] Key type lgssc unregistered [ 4772.567125] LNet: 155217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4772.576650] LNetError: 155217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4773.611154] LNet: Removed LNI 192.168.202.145@tcp [ 4774.705568] Key type .llcrypt unregistered [ 4774.709638] Key type ._llcrypt unregistered [ 4815.445105] Key type ._llcrypt registered [ 4815.446305] Key type .llcrypt registered [ 4815.531383] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 4816.855917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4816.883038] alg: No test for adler32 (adler32-zlib) [ 4817.985940] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 4818.284209] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 4819.952189] Key type lgssc registered [ 4821.578332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4831.424414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4831.445666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4832.966390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4837.885345] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4842.466472] LustreError: 156757:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4842.479927] LustreError: 156757:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4847.589801] LustreError: 156756:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4848.101978] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4852.717818] LustreError: 156757:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4852.826717] LustreError: 156785:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4852.848518] LustreError: 156785:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4852.867969] LustreError: 156785:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 4853.193674] Lustre: server umount lustre-MDT0000 complete [ 4873.466528] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 4884.139452] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 4919.765709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4920.067363] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4920.117840] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4920.266392] Lustre: lustre-MDT0000: new disk, initializing [ 4920.391212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4920.407613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4925.240762] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4936.217026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4936.448753] Lustre: 160092:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4936.561902] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4936.569425] Lustre: Skipped 1 previous similar message [ 4936.690683] Lustre: lustre-MDT0001: new disk, initializing [ 4936.810540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4936.888368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4936.908784] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4941.331187] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4951.739806] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4957.369116] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4964.397291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4964.643254] Lustre: lustre-OST0000: new disk, initializing [ 4964.646901] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4964.652473] Lustre: 162060:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4964.735096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4971.720042] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4972.656565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4972.669417] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4972.798780] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4982.341317] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4987.285247] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4988.793694] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 10:17:37 (1790777857) [ 5213.152856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5213.170088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5213.193421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5217.135720] Lustre: server umount lustre-MDT0000 complete [ 5218.788523] LustreError: 160104:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5218.812671] LustreError: 160104:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5221.182984] LustreError: 160084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790778091 with bad export cookie 14580888807571416110 [ 5221.189058] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5221.193871] LustreError: 160084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5223.906042] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5223.908039] LustreError: 163841:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5223.922036] Lustre: Skipped 2 previous similar messages [ 5223.942805] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5223.949515] Lustre: Skipped 2 previous similar messages [ 5226.913606] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5227.583629] Lustre: server umount lustre-MDT0001 complete [ 5237.548463] Lustre: server umount lustre-OST0000 complete [ 5247.165447] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 5257.742853] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 5293.769661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5293.985507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5294.014334] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5294.083353] Lustre: lustre-MDT0000: new disk, initializing [ 5294.148294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5294.171300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5297.966676] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5308.767323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5308.910135] Lustre: 166719:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5308.954305] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5308.958324] Lustre: Skipped 1 previous similar message [ 5309.036482] Lustre: lustre-MDT0001: new disk, initializing [ 5309.139965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5309.170993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5309.175437] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5314.172827] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5326.932773] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5333.179664] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5341.990495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5342.459067] Lustre: lustre-OST0000: new disk, initializing [ 5342.465460] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5342.477442] Lustre: 168688:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5342.618242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5347.929402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5347.936598] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5348.073049] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5349.492078] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5359.097738] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5364.460730] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5364.688815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5370.079305] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5370.424473] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5373.921185] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5373.922260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5373.930446] LustreError: Skipped 1 previous similar message [ 5373.936590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5373.943746] Lustre: Skipped 1 previous similar message [ 5373.974785] Lustre: Skipped 1 previous similar message [ 5378.790500] Lustre: server umount lustre-OST0000 complete [ 5384.162423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5384.175563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5384.183175] Lustre: Skipped 1 previous similar message [ 5388.729485] Lustre: server umount lustre-MDT0000 complete [ 5389.281809] LustreError: 166731:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5389.306768] LustreError: 166731:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5392.597442] LustreError: 166713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790778263 with bad export cookie 14580888807571574940 [ 5392.599447] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5392.613177] LustreError: 166713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5393.185390] Lustre: server umount lustre-MDT0001 complete [ 5401.890627] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:24:30 (1790778270) [ 5416.873797] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 5425.955478] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 5465.093849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5465.317498] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5465.336876] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5465.410189] Lustre: lustre-MDT0000: new disk, initializing [ 5465.456945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5465.465138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5470.532313] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5482.620038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5482.803140] Lustre: 173447:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5483.098714] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5487.555304] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5497.569975] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5503.847819] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5510.429497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5510.827836] Lustre: lustre-OST0000: new disk, initializing [ 5510.830443] Lustre: Skipped 1 previous similar message [ 5510.835127] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5510.838836] Lustre: Skipped 2 previous similar messages [ 5510.845283] Lustre: 175424:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5517.674900] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5518.955985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5518.974287] Lustre: Skipped 1 previous similar message [ 5518.988441] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5519.122370] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5526.809283] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5534.689721] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5534.696891] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5534.707026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5534.723482] Lustre: Skipped 2 previous similar messages [ 5540.252805] Lustre: server umount lustre-OST0000 complete [ 5544.422275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5544.440469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5550.679036] Lustre: server umount lustre-MDT0000 complete [ 5554.642687] LustreError: 174246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790778425 with bad export cookie 14580888807571575822 [ 5554.649907] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5554.661742] LustreError: 174246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5555.043451] Lustre: server umount lustre-MDT0001 complete [ 5564.308771] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:27:12 (1790778432) [ 5581.347675] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 5593.919602] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 5630.466087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5630.622415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5630.648713] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5630.713279] Lustre: lustre-MDT0000: new disk, initializing [ 5630.779882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5630.784749] Lustre: Skipped 2 previous similar messages [ 5630.800940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5634.943366] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5644.724374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5644.860645] Lustre: 179745:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5645.057302] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5649.505177] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5658.953979] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5665.886135] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5673.288326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5673.791830] Lustre: 181723:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5680.484372] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5681.222725] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5681.322932] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5691.530993] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5702.113426] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5702.116518] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5702.123013] LustreError: Skipped 1 previous similar message [ 5702.127949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5702.142599] Lustre: Skipped 2 previous similar messages [ 5702.156087] Lustre: Skipped 8 previous similar messages [ 5706.096425] Lustre: server umount lustre-OST0000 complete [ 5711.846073] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5715.413560] Lustre: server umount lustre-MDT0000 complete [ 5717.477049] LustreError: 179752:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5717.499986] LustreError: 179752:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5718.974812] LustreError: 179738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790778589 with bad export cookie 14580888807571576907 [ 5718.982136] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5719.359222] Lustre: server umount lustre-MDT0001 complete [ 5730.005886] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 5732.596907] Key type lgssc unregistered [ 5732.812750] LNet: 183580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5732.819621] LNetError: 183580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5732.836392] LNet: Removed LNI 192.168.202.145@tcp [ 5733.448278] Key type .llcrypt unregistered [ 5733.451335] Key type ._llcrypt unregistered [ 5762.476370] Key type ._llcrypt registered [ 5762.483223] Key type .llcrypt registered [ 5762.606368] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 5763.629589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5763.873091] alg: No test for adler32 (adler32-zlib) [ 5765.151846] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 5765.592314] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 5767.480241] Key type lgssc registered [ 5768.848619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5776.726961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5776.744345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5778.318756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5782.915341] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5788.641552] LustreError: 185012:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5788.663429] LustreError: 185012:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5791.129810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5791.467233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5796.919874] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5804.981564] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5810.700548] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5818.764540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5819.290258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5823.446669] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5826.590305] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5835.125683] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5854.177351] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5854.188153] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5854.216445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5857.258603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5857.259617] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5859.968330] Lustre: server umount lustre-OST0000 complete [ 5867.494630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5867.497527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5867.511555] Lustre: Skipped 1 previous similar message [ 5867.524551] Lustre: Skipped 2 previous similar messages [ 5870.514064] Lustre: server umount lustre-MDT0000 complete [ 5872.611894] LustreError: 185741:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5872.645749] LustreError: 185741:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5874.244539] LustreError: 184992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790778744 with bad export cookie 11128101842082316928 [ 5874.257964] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5874.559678] Lustre: server umount lustre-MDT0001 complete [ 5888.265104] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 5891.298591] Key type lgssc unregistered [ 5891.620217] LNet: 188568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5891.634328] LNetError: 188568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5891.668680] LNet: Removed LNI 192.168.202.145@tcp [ 5892.539129] Key type .llcrypt unregistered [ 5892.541322] Key type ._llcrypt unregistered [ 5917.746531] Key type ._llcrypt registered [ 5917.749706] Key type .llcrypt registered [ 5917.878083] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 5919.195586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5919.239576] alg: No test for adler32 (adler32-zlib) [ 5920.280388] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 5920.531238] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 5922.248295] Key type lgssc registered [ 5923.407029] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5930.727606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5930.741908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5932.141380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5936.410442] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5941.731197] LustreError: 189999:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5941.752297] LustreError: 189999:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5944.462345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5944.848768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5949.885575] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5957.844907] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5963.926212] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5971.022929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5971.391443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5976.519856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5977.514648] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5985.287887] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5997.027629] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5997.046928] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5997.068081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5998.400882] Lustre: lustre-OST0000: Not available for connect from 192.168.202.45@tcp (stopping) [ 6000.107609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6000.111266] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6000.113442] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6000.119200] Lustre: Skipped 1 previous similar message [ 6001.524301] Lustre: server umount lustre-OST0000 complete [ 6009.103876] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff89d252648800.ost_server_uuid 50 [ 6010.649148] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6024.674050] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6024.683527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6024.687321] Lustre: Skipped 1 previous similar message [ 6029.800881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6029.830875] Lustre: Skipped 1 previous similar message [ 6030.959243] Lustre: server umount lustre-MDT0000 complete [ 6034.531931] LustreError: 189981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790778905 with bad export cookie 6707767763624332192 [ 6034.550896] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6034.554849] LustreError: 189981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6034.762930] Lustre: server umount lustre-MDT0001 complete [ 6042.088885] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:35:10 (1790778910) [ 6048.244047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6048.802946] LustreError: 193916:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6048.918069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6054.370619] LustreError: 193917:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6054.777666] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6058.465833] LustreError: 193916:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6063.605123] LustreError: 193917:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6066.959077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6067.533429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6073.525856] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6082.503612] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6088.015678] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6094.748167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6095.049199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6100.572468] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6108.665994] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6115.300182] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6115.563588] Lustre: lustre-OST0001: new disk, initializing [ 6115.566753] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6115.577660] Lustre: 196758:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6115.677925] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6118.785754] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6121.663279] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6122.660341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6122.665741] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6122.756798] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6130.766754] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6133.225519] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6133.229093] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6133.244333] Lustre: Skipped 1 previous similar message [ 6133.245410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6139.140932] Lustre: server umount lustre-OST0000 complete [ 6141.762829] LustreError: 195741:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6147.419382] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6147.693789] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6153.024101] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6153.250592] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6158.818185] LustreError: 195740:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6158.847201] LustreError: 195740:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 6169.064353] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6169.072936] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6169.075986] LustreError: Skipped 1 previous similar message [ 6169.077066] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6169.077071] Lustre: Skipped 4 previous similar messages [ 6169.097740] Lustre: Skipped 2 previous similar messages [ 6173.606832] Lustre: server umount lustre-OST0001 complete [ 6179.297664] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6179.330219] Lustre: Skipped 1 previous similar message [ 6192.096345] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6192.389166] Lustre: server umount lustre-MDT0000 complete [ 6193.654814] LustreError: 193913:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6193.700189] LustreError: 193913:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 6196.288339] LustreError: 193896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790779067 with bad export cookie 6707767763624333270 [ 6196.298600] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6196.302378] LustreError: 193896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6196.747993] Lustre: server umount lustre-MDT0001 complete [ 6220.440341] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:38:09 (1790779089) [ 6226.659598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6227.033867] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6227.086405] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6227.433850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6232.507487] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6242.884794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6243.035740] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6243.101674] Lustre: 200201:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6250.642834] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6259.454192] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6265.639954] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6273.021069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6273.212606] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6279.295272] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6280.684118] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6287.943268] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6294.612262] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6294.817832] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6295.059476] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6295.069454] Lustre: Skipped 2 previous similar messages [ 6301.253907] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6303.506865] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6308.437720] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6313.962441] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6313.986776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6313.991087] Lustre: Skipped 8 previous similar messages [ 6316.955964] Lustre: server umount lustre-OST0000 complete [ 6319.088190] LustreError: 202054:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6319.113757] LustreError: 202054:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6334.435800] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6334.454369] LustreError: Skipped 1 previous similar message [ 6337.792396] Lustre: server umount lustre-OST0001 complete [ 6349.449241] Lustre: server umount lustre-MDT0000 complete [ 6352.888765] LustreError: 200193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790779223 with bad export cookie 6707767763624334677 [ 6352.894776] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6352.902980] LustreError: 200193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6353.336177] Lustre: server umount lustre-MDT0001 complete [ 6374.477272] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:40:43 (1790779243) [ 6389.897813] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 6400.611790] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 6431.810810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6432.018995] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6432.048808] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6432.153780] Lustre: lustre-MDT0000: new disk, initializing [ 6432.247065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6432.274543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6436.737108] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6446.178298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6446.266307] Lustre: 208295:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6446.313356] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6446.316951] Lustre: Skipped 1 previous similar message [ 6446.402885] Lustre: lustre-MDT0001: new disk, initializing [ 6446.537048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6446.545577] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6450.780448] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6461.253269] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6468.483185] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6475.835426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6476.137855] Lustre: lustre-OST0000: new disk, initializing [ 6476.142746] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6476.150997] Lustre: 210265:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6482.981609] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6485.595698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6485.607807] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6485.748075] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6493.417375] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6497.826399] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6498.047552] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6502.900654] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6503.122997] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6506.471640] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6506.471894] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6506.475163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6506.475173] Lustre: Skipped 5 previous similar messages [ 6506.484349] Lustre: Skipped 6 previous similar messages [ 6506.499159] LustreError: Skipped 1 previous similar message [ 6519.265415] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6519.437982] Lustre: server umount lustre-OST0000 complete [ 6523.375082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6537.184157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6537.295493] Lustre: server umount lustre-MDT0000 complete [ 6541.900379] LustreError: 209089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790779412 with bad export cookie 6707767763624336455 [ 6541.903880] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6541.913141] LustreError: 209089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6542.274871] Lustre: server umount lustre-MDT0001 complete [ 6552.025550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6552.668254] LustreError: 212485:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6552.685840] LustreError: 212485:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 6557.628516] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6566.199319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6570.309895] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6578.655996] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6584.213112] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6590.885326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6597.792717] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6604.709926] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6610.195472] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6610.435060] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6615.473720] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6615.841446] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6618.080924] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6618.082160] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6618.094741] LustreError: Skipped 1 previous similar message [ 6618.116904] Lustre: Skipped 3 previous similar messages [ 6624.341273] Lustre: server umount lustre-OST0000 complete [ 6630.829189] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6635.680283] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6664.330049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6670.751276] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6674.022477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6674.034231] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6674.036014] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6677.502520] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6684.674628] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec [ 6689.767949] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6689.776646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6689.777180] LustreError: Skipped 1 previous similar message [ 6689.786192] Lustre: Skipped 17 previous similar messages [ 6703.072253] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6703.185732] Lustre: server umount lustre-OST0000 complete [ 6713.105881] Lustre: server umount lustre-MDT0000 complete [ 6716.020161] LustreError: 212466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790779586 with bad export cookie 6707767763624337344 [ 6716.026743] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6716.028210] LustreError: 212466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6716.343984] Lustre: server umount lustre-MDT0001 complete [ 6723.923794] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:46:32 (1790779592) [ 6730.083364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6730.711276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6730.718244] Lustre: Skipped 6 previous similar messages [ 6735.492656] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6743.520337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6748.191560] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6754.210075] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6761.221557] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6769.253376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6779.533596] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6787.810427] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6792.773344] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6793.049609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6797.988274] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6798.237215] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6804.675674] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6804.864027] Lustre: lustre-OST0001: new disk, initializing [ 6804.868199] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6804.877149] Lustre: 221823:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6812.090935] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6814.297628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6814.315195] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6814.423771] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6821.641702] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6826.333780] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6826.544871] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6831.118179] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6831.303260] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6835.169145] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6835.170149] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6835.182319] LustreError: Skipped 2 previous similar messages [ 6835.192300] Lustre: Skipped 4 previous similar messages [ 6839.291487] Lustre: server umount lustre-OST0001 complete [ 6840.297722] LustreError: 220276:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6840.323764] LustreError: 220276:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 6844.839081] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6845.029940] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6848.808185] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6849.015571] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6877.316606] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6882.804320] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6882.829399] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6882.848214] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6882.863301] Lustre: Skipped 2 previous similar messages [ 6884.308710] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6891.524836] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6893.139054] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6901.301319] Lustre: server umount lustre-OST0001 complete [ 6924.528700] Lustre: server umount lustre-OST0000 complete [ 6936.031135] Lustre: server umount lustre-MDT0000 complete [ 6939.527613] LustreError: 218433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790779810 with bad export cookie 6707767763624338303 [ 6939.528859] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6939.537525] LustreError: 218433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6945.956858] Lustre: server umount lustre-MDT0001 complete [ 6969.302525] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:50:38 (1790779838) [ 6975.846653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6976.344471] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6976.374229] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6981.767626] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 6991.093414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6991.190359] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6991.210873] Lustre: 227066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6996.427868] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7003.438123] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7009.373275] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7016.666500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7016.812553] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7023.292865] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7032.194753] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7042.286123] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7042.400761] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7049.947235] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7056.720470] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7058.335513] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7062.950931] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7063.105422] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7067.616471] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7067.784584] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7073.289446] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff89d248b40000.ost_server_uuid 50 [ 7074.742615] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff89d248b40000.ost_server_uuid in IDLE state after 0 sec [ 7075.856894] Lustre: Permanently deactivating lustre-OST0001 [ 7075.873933] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7079.124992] Lustre: Permanently reactivating lustre-OST0001 [ 7086.567985] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7086.579991] Lustre: Skipped 20 previous similar messages [ 7089.153786] Lustre: server umount lustre-OST0001 complete [ 7092.709978] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7092.720076] LustreError: Skipped 6 previous similar messages [ 7092.722050] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7092.732841] Lustre: Skipped 9 previous similar messages [ 7098.904120] Lustre: server umount lustre-OST0000 complete [ 7117.285723] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7117.646423] Lustre: server umount lustre-MDT0000 complete [ 7121.025776] LustreError: 227809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790779991 with bad export cookie 6707767763624339598 [ 7121.031820] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7121.339895] Lustre: server umount lustre-MDT0001 complete [ 7148.920763] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:53:37 (1790780017) [ 7157.789358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7158.031875] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7158.070440] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7158.084886] Lustre: Skipped 5 previous similar messages [ 7163.906542] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7172.576893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7172.648224] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7172.687828] Lustre: 233762:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7177.585967] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7185.402392] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7190.749390] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7198.642356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7198.847179] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7198.874729] Lustre: Permanently deactivating lustre-OST0000 [ 7198.888741] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7204.907137] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7205.028610] LustreError: 235923:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -108 [ 7207.271897] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7215.720122] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7223.834510] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7224.045029] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7231.402347] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7233.196578] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7238.384483] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7240.221264] Lustre: Permanently reactivating lustre-OST0000 [ 7240.242745] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7240.250934] Lustre: Skipped 2 previous similar messages [ 7242.331651] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7242.337958] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7242.348400] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7242.409176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7242.427361] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7269.345039] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7269.736914] Lustre: server umount lustre-OST0001 complete [ 7289.313879] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7289.598709] Lustre: server umount lustre-OST0000 complete [ 7300.658288] Lustre: server umount lustre-MDT0000 complete [ 7304.890118] LustreError: 233754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790780175 with bad export cookie 6707767763624341271 [ 7304.897436] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7304.900763] LustreError: 233754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7305.366806] Lustre: server umount lustre-MDT0001 complete [ 7318.567579] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 7321.916610] Key type lgssc unregistered [ 7322.399651] LNet: 238645:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7322.407979] LNetError: 238645:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7322.465359] LNet: Removed LNI 192.168.202.145@tcp [ 7323.473627] Key type .llcrypt unregistered [ 7323.475095] Key type ._llcrypt unregistered [ 7346.474309] Key type ._llcrypt registered [ 7346.475954] Key type .llcrypt registered [ 7346.619405] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 7348.033563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7348.140158] alg: No test for adler32 (adler32-zlib) [ 7349.330655] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 7349.594691] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 7351.320525] Key type lgssc registered [ 7352.836776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7363.250331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7363.271636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7364.726889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7369.742516] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7374.307041] LustreError: 240129:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7374.346101] LustreError: 240129:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7379.429946] LustreError: 240128:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7380.107682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7380.228082] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7380.300199] Lustre: 240116:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7380.311903] Lustre: Permanently deactivating lustre-MDT0001 [ 7380.329549] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7380.564176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7382.740577] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7382.745601] LustreError: 240159:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7382.759508] LustreError: 240159:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7382.773248] LustreError: 240159:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 1, failed: rc = -108 [ 7385.838632] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7393.843819] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7399.697442] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7406.746399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7407.197032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7409.269784] LustreError: 241971:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7409.296896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7413.633409] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7414.759924] LustreError: 241968:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7419.875706] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7419.887585] LustreError: 242098:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7426.064903] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7427.082552] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7427.235657] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7432.714894] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7432.725520] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7435.073690] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7444.126437] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7448.088170] Lustre: Permanently reactivating lustre-MDT0001 [ 7448.098110] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7448.108518] Lustre: Skipped 2 previous similar messages [ 7454.042761] Lustre: lustre-MDT0001: Client fd863916-4be3-439d-b639-295a8dc81c3f (at 192.168.202.45@tcp) reconnecting [ 7461.893407] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7464.125786] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7471.460585] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7471.676661] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7475.036842] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7475.256983] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7477.648458] Lustre: Permanently deactivating lustre-MDT0001 [ 7477.659881] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7477.664784] Lustre: Skipped 2 previous similar messages [ 7485.632123] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7493.314640] Lustre: server umount lustre-MDT0000 complete [ 7497.187561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7497.192745] LustreError: 240125:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7497.211242] Lustre: Skipped 1 previous similar message [ 7497.234067] LustreError: 240125:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 7498.006551] LustreError: 244611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790780368 with bad export cookie 5623449301395490194 [ 7498.011959] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7498.029238] LustreError: 244611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7498.483953] Lustre: server umount lustre-MDT0001 complete [ 7502.076170] Lustre: server umount lustre-OST0000 complete [ 7514.128923] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:59:42 (1790780382) [ 7525.687450] Lustre: server umount lustre-OST0001 complete [ 7533.980388] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 7543.939683] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 7582.990276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7583.311264] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7583.322851] Lustre: Skipped 2 previous similar messages [ 7583.363392] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7583.443855] Lustre: lustre-MDT0000: new disk, initializing [ 7583.572588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7583.640828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7589.019575] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7600.990064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7601.149684] Lustre: 248385:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7601.206616] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7601.210322] Lustre: Skipped 1 previous similar message [ 7601.290542] Lustre: lustre-MDT0001: new disk, initializing [ 7601.376828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7601.422112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7601.431165] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7606.485200] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7616.906686] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7622.919747] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7629.770407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7630.039465] Lustre: lustre-OST0000: new disk, initializing [ 7630.042694] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7630.048503] Lustre: 250364:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7630.182864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7636.293145] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7637.127282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7637.134871] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7637.230244] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7646.110423] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7648.368245] LustreError: 248392:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7657.167054] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7657.309448] Lustre: lustre-OST0001: new disk, initializing [ 7657.315751] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7657.320614] Lustre: 251480:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7658.386311] LustreError: 248392:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7658.403762] LustreError: 251247:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7663.403063] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7665.227771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7665.245517] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7665.394373] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7668.488731] LustreError: 251247:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7674.577688] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7676.327798] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7680.998841] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7681.011322] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7681.012946] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7681.019244] LustreError: Skipped 1 previous similar message [ 7681.053662] Lustre: Skipped 3 previous similar messages [ 7684.528429] Lustre: server umount lustre-OST0001 complete [ 7686.125037] LustreError: 250778:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7686.165049] LustreError: 250778:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7704.037162] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7704.038297] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7704.051150] LustreError: Skipped 1 previous similar message [ 7704.052222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7704.052228] Lustre: Skipped 1 previous similar message [ 7704.084319] Lustre: Skipped 1 previous similar message [ 7706.595933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7706.614432] Lustre: Skipped 2 previous similar messages [ 7708.860128] Lustre: server umount lustre-OST0000 complete [ 7714.279341] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7714.291381] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7714.310107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7721.960786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7721.969079] Lustre: Skipped 3 previous similar messages [ 7728.096203] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7728.335971] Lustre: server umount lustre-MDT0000 complete [ 7732.117261] LustreError: 250363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790780602 with bad export cookie 5623449301395493008 [ 7732.122171] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7732.122843] LustreError: 250363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7738.721427] Lustre: server umount lustre-MDT0001 complete [ 7749.789843] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 7752.575303] Key type lgssc unregistered [ 7752.831521] LNet: 253517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7752.836831] LNetError: 253517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7752.850272] LNet: Removed LNI 192.168.202.145@tcp [ 7753.480909] Key type .llcrypt unregistered [ 7753.484975] Key type ._llcrypt unregistered [ 7797.087872] Key type ._llcrypt registered [ 7797.090870] Key type .llcrypt registered [ 7797.283698] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 7798.382305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7798.445129] alg: No test for adler32 (adler32-zlib) [ 7799.679161] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 7800.150506] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 7801.832532] Key type lgssc registered [ 7804.329898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7812.141863] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7812.158198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7813.387016] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7813.438542] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7813.640635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7818.865498] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7830.218490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7830.425588] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7830.483770] Lustre: 255737:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7830.727014] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7836.907553] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7844.892570] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7851.170787] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7858.513657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7858.870064] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7859.284121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7867.163819] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7868.053600] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7877.489537] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7933.937190] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7933.939169] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7933.946679] LustreError: Skipped 1 previous similar message [ 7933.959552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7933.970929] Lustre: Skipped 1 previous similar message [ 7933.994134] Lustre: Skipped 1 previous similar message [ 7936.905959] Lustre: server umount lustre-OST0000 complete [ 7942.370092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7955.974953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7956.410922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7962.807705] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 7971.121481] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7973.805636] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7973.815965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7973.832236] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8014.818440] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8014.825585] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8014.829131] LustreError: Skipped 1 previous similar message [ 8014.832365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8014.870259] Lustre: Skipped 1 previous similar message [ 8018.122954] Lustre: server umount lustre-OST0000 complete [ 8025.066716] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8025.069604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8025.085641] Lustre: Skipped 1 previous similar message [ 8025.096574] Lustre: Skipped 2 previous similar messages [ 8028.809923] Lustre: server umount lustre-MDT0000 complete [ 8032.584878] LustreError: 255729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790780903 with bad export cookie 16476620502511316057 [ 8032.588846] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8032.593991] LustreError: 255729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8032.946543] Lustre: server umount lustre-MDT0001 complete [ 8045.824438] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 8048.989504] Key type lgssc unregistered [ 8049.329866] LNet: 261165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8049.335352] LNetError: 261165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8049.358357] LNet: Removed LNI 192.168.202.145@tcp [ 8050.327260] Key type .llcrypt unregistered [ 8050.328964] Key type ._llcrypt unregistered [ 8078.265326] Key type ._llcrypt registered [ 8078.271260] Key type .llcrypt registered [ 8078.411847] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 8079.654364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8079.874751] alg: No test for adler32 (adler32-zlib) [ 8080.975454] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 8081.301286] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 8083.048239] Key type lgssc registered [ 8084.546708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8092.514758] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8092.543211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8094.054573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8098.569517] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8103.398598] LustreError: 262597:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8103.411180] LustreError: 262597:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8107.778325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8108.273807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8112.933431] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8121.245644] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8127.919080] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8134.899641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8135.311172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8138.422273] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8141.664052] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8149.679872] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8178.659941] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8178.676588] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8178.685599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8178.719864] Lustre: Skipped 1 previous similar message [ 8178.738763] Lustre: Skipped 1 previous similar message [ 8183.780069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8183.782857] Lustre: Skipped 1 previous similar message [ 8184.397491] Lustre: server umount lustre-OST0000 complete [ 8188.898887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8188.902262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8188.909600] Lustre: Skipped 1 previous similar message [ 8188.928600] Lustre: Skipped 1 previous similar message [ 8194.020306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8194.026148] Lustre: Skipped 1 previous similar message [ 8194.218685] Lustre: server umount lustre-MDT0000 complete [ 8197.413763] LustreError: 262579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790781068 with bad export cookie 15204361504788794590 [ 8197.429062] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8197.432340] LustreError: 262579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8197.854981] Lustre: server umount lustre-MDT0001 complete [ 8210.801818] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 8214.610754] Key type lgssc unregistered [ 8215.171079] LNet: 266902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8215.187337] LNetError: 266902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8215.211868] LNet: Removed LNI 192.168.202.145@tcp [ 8216.381571] Key type .llcrypt unregistered [ 8216.387211] Key type ._llcrypt unregistered [ 8236.097560] Key type ._llcrypt registered [ 8236.099468] Key type .llcrypt registered [ 8236.242803] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 8237.955344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8238.247356] alg: No test for adler32 (adler32-zlib) [ 8239.556704] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 8239.880551] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 8241.712218] Key type lgssc registered [ 8242.395590] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8250.676737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8250.694634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8252.036256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8258.687344] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8261.614360] LustreError: 268140:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8261.627502] LustreError: 268140:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8266.730658] LustreError: 268141:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8268.388888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8268.799328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8273.098610] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8279.227537] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8284.958151] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8291.148537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8291.551766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8296.732518] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8298.460333] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8306.347182] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8319.467832] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8319.478453] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8319.498187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8322.533583] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8322.535768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8322.536074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8322.574884] Lustre: Skipped 1 previous similar message [ 8325.206809] Lustre: server umount lustre-OST0000 complete [ 8330.209873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8330.215153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8330.225888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8332.779676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8332.791806] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8335.850442] Lustre: server umount lustre-MDT0000 complete [ 8337.892800] LustreError: 268140:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8337.909703] LustreError: 268140:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8339.638356] LustreError: 268120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790781210 with bad export cookie 8528057969296356294 [ 8339.639820] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8339.644658] LustreError: 268120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8339.934256] Lustre: server umount lustre-MDT0001 complete [ 8353.816393] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 8356.799888] Key type lgssc unregistered [ 8357.050965] LNet: 271817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8357.081901] LNetError: 271817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8357.103396] LNet: Removed LNI 192.168.202.145@tcp [ 8358.015833] Key type .llcrypt unregistered [ 8358.019378] Key type ._llcrypt unregistered [ 8385.303631] Key type ._llcrypt registered [ 8385.305819] Key type .llcrypt registered [ 8385.455372] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 8387.300192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8387.348502] alg: No test for adler32 (adler32-zlib) [ 8388.484972] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 8388.856820] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 8390.592576] Key type lgssc registered [ 8392.343653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8402.184989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8402.215228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8403.936720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8409.047377] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8414.182398] LustreError: 273250:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8414.204381] LustreError: 273250:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8419.316049] LustreError: 273251:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8420.145425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8420.720876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8427.193577] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8436.458680] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8442.718854] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8449.745487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8450.133974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8457.859207] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8466.584543] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8473.742183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8501.732296] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8501.735465] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8501.759157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8504.822295] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8504.831762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8504.840399] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8504.878188] Lustre: Skipped 1 previous similar message [ 8507.831173] Lustre: server umount lustre-OST0000 complete [ 8512.993457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8513.002943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8513.025803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8515.043700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8515.045759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8515.067090] Lustre: Skipped 1 previous similar message [ 8517.987463] Lustre: server umount lustre-MDT0000 complete [ 8520.169788] LustreError: 273246:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8520.217320] LustreError: 273246:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8522.066685] LustreError: 273230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790781392 with bad export cookie 17516913982438477064 [ 8522.069368] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8522.073833] LustreError: 273230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8522.802893] Lustre: server umount lustre-MDT0001 complete [ 8535.932090] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 8539.047936] Key type lgssc unregistered [ 8539.387727] LNet: 277577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8539.398394] LNetError: 277577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8539.419137] LNet: Removed LNI 192.168.202.145@tcp [ 8540.399323] Key type .llcrypt unregistered [ 8540.406787] Key type ._llcrypt unregistered [ 8560.111968] Key type ._llcrypt registered [ 8560.115443] Key type .llcrypt registered [ 8560.307212] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 8562.239383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8562.297251] alg: No test for adler32 (adler32-zlib) [ 8563.384937] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 8563.610427] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 8565.353826] Key type lgssc registered [ 8566.746778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8577.514935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8577.543377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8579.422360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8585.621868] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8589.799637] LustreError: 278830:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8589.812428] LustreError: 278830:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8594.916425] LustreError: 278832:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8595.448339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8596.067934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8600.917888] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8609.045844] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8614.922818] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8622.124077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8622.587667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8624.700489] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8629.871347] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8637.778623] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8655.329365] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8655.338367] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8655.361516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8656.369384] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8656.372845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8656.379355] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8656.412181] Lustre: Skipped 1 previous similar message [ 8659.741822] Lustre: server umount lustre-OST0000 complete [ 8666.594090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8666.597349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8666.610149] Lustre: Skipped 1 previous similar message [ 8666.624456] Lustre: Skipped 1 previous similar message [ 8671.172558] Lustre: server umount lustre-MDT0000 complete [ 8671.723460] LustreError: 278840:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8671.750781] LustreError: 278840:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8675.743226] LustreError: 278797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790781546 with bad export cookie 2239091638626216764 [ 8675.759739] LustreError: 278797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8675.762590] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8676.228431] Lustre: server umount lustre-MDT0001 complete [ 8690.052045] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 8693.728757] Key type lgssc unregistered [ 8694.122203] LNet: 282532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8694.129248] LNetError: 282532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8694.143200] LNet: Removed LNI 192.168.202.145@tcp [ 8695.511369] Key type .llcrypt unregistered [ 8695.513995] Key type ._llcrypt unregistered [ 8709.941924] Key type ._llcrypt registered [ 8709.944337] Key type .llcrypt registered [ 8710.103310] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8730.616541] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 8748.701565] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 8750.525294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8750.567031] alg: No test for adler32 (adler32-zlib) [ 8751.881770] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 8752.369199] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 8754.200464] Key type lgssc registered [ 8755.966674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8797.005515] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8797.039363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8798.281659] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8798.327319] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8798.379314] Lustre: lustre-MDT0000: new disk, initializing [ 8798.429164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8798.438517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8803.095640] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8815.095041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8815.265349] Lustre: 286104:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8815.390041] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8815.403463] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8815.404628] Lustre: Skipped 1 previous similar message [ 8815.495306] Lustre: lustre-MDT0001: new disk, initializing [ 8815.624280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8815.693958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8815.707551] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8822.144535] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8833.369439] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8841.166706] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8848.465620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8848.733556] Lustre: lustre-OST0000: new disk, initializing [ 8848.736941] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8848.748353] Lustre: 288070:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8848.849615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8855.160285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8855.177029] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8855.301887] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8856.853881] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8868.042178] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8873.593360] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8873.874796] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8879.904672] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8880.173733] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8886.248182] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8886.254465] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8886.259018] LustreError: Skipped 1 previous similar message [ 8886.260183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8886.283476] Lustre: Skipped 1 previous similar message [ 8888.374767] Lustre: server umount lustre-OST0000 complete [ 8896.482069] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8896.492631] Lustre: Skipped 1 previous similar message [ 8896.493032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8896.504959] Lustre: Skipped 2 previous similar messages [ 8901.604451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8901.614271] Lustre: Skipped 1 previous similar message [ 8906.724084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8906.731148] Lustre: Skipped 1 previous similar message [ 8907.232180] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8907.568816] Lustre: server umount lustre-MDT0000 complete [ 8912.832343] LustreError: 286094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790781783 with bad export cookie 16151219703446199423 [ 8912.835388] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8912.843916] LustreError: 286094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8913.183678] Lustre: server umount lustre-MDT0001 complete [ 8923.922518] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:23:12 (1790781792) [ 8930.689517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8930.939934] LustreError: 290491:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8930.954000] LustreError: 290491:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8931.008219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8936.421536] LustreError: 290492:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8936.764912] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8940.523062] LustreError: 290491:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8945.633967] LustreError: 290492:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8949.505749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8950.260348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8955.997216] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8963.264701] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8969.921780] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8979.817278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8980.029315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8987.667825] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 8995.748646] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9010.660406] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9010.661040] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9010.670497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9010.675471] Lustre: Skipped 1 previous similar message [ 9015.783585] LustreError: 292984:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9015.795586] LustreError: 292984:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9015.939422] Lustre: server umount lustre-OST0000 complete [ 9020.908870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9023.518382] Lustre: server umount lustre-MDT0000 complete [ 9026.024815] LustreError: 290492:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9026.049636] LustreError: 290492:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9027.817266] LustreError: 290471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790781898 with bad export cookie 16151219703446200305 [ 9027.823927] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9027.826117] LustreError: 290471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9028.363470] Lustre: server umount lustre-MDT0001 complete [ 9044.198748] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 9047.259736] Key type lgssc unregistered [ 9047.588762] LNet: 294052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9047.593805] LNetError: 294052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9047.612457] LNet: Removed LNI 192.168.202.145@tcp [ 9048.486151] Key type .llcrypt unregistered [ 9048.489697] Key type ._llcrypt unregistered [ 9078.592318] Key type ._llcrypt registered [ 9078.595389] Key type .llcrypt registered [ 9078.739304] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 9094.731296] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 9095.650226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9095.843911] alg: No test for adler32 (adler32-zlib) [ 9097.004935] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 9097.412269] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 9099.144319] Key type lgssc registered [ 9100.943935] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9151.550816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9151.582522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9152.766393] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9152.786380] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9152.834065] Lustre: lustre-MDT0000: new disk, initializing [ 9152.907228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9152.920933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9157.975529] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9170.212273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9170.373496] Lustre: 297892:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9170.438762] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9170.443743] Lustre: Skipped 1 previous similar message [ 9170.521683] Lustre: lustre-MDT0001: new disk, initializing [ 9170.606985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9170.641237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9170.659443] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9176.144171] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9187.882710] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9195.079369] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9203.116181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9203.508515] Lustre: lustre-OST2710: new disk, initializing [ 9203.512419] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9203.539904] Lustre: 299865:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9203.735679] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9211.024156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9211.046233] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9211.239532] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9211.624851] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9221.830803] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9229.237496] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9229.466843] Lustre: lustre-OST03e8: new disk, initializing [ 9229.472188] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9229.486796] Lustre: 300927:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9229.632841] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9235.542835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9235.559070] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9235.701850] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9236.658944] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9246.824688] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9254.168468] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9254.435728] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9258.953594] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9259.039356] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9264.810577] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9265.060485] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9270.764101] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9271.186599] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9278.434011] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9278.441137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9278.458976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9282.017402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9282.018057] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9282.030725] Lustre: Skipped 1 previous similar message [ 9283.981324] Lustre: server umount lustre-MDT0000 complete [ 9288.078457] LustreError: 297885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790782158 with bad export cookie 13809088736183499459 [ 9288.086122] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9288.094446] LustreError: 297885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9288.572231] Lustre: server umount lustre-MDT0001 complete [ 9302.897447] Lustre: server umount lustre-OST2710 complete [ 9316.786473] Lustre: server umount lustre-OST03e8 complete [ 9339.915422] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 9355.125335] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 9395.300590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9395.699660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9395.761854] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9395.926250] Lustre: lustre-MDT0000: new disk, initializing [ 9396.021038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9396.048887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9403.104946] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9414.484981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9414.770647] Lustre: 306203:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9414.850630] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9414.853924] Lustre: Skipped 1 previous similar message [ 9414.904210] Lustre: lustre-MDT0001: new disk, initializing [ 9414.994130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9415.020318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9415.030561] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9420.449518] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9430.366838] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9436.723034] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9444.041873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9444.292265] Lustre: lustre-OST0000: new disk, initializing [ 9444.295618] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9444.301886] Lustre: 308172:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9444.358502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9450.629144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9450.650452] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9450.808226] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9451.484596] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9460.838967] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9465.196112] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9465.324554] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9470.031551] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9470.234110] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9476.579758] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9476.584435] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9476.592546] LustreError: Skipped 1 previous similar message [ 9476.593475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9476.606385] Lustre: Skipped 2 previous similar messages [ 9476.624857] Lustre: Skipped 4 previous similar messages [ 9478.418112] Lustre: server umount lustre-OST0000 complete [ 9486.825571] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9486.834850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9486.848838] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9486.862541] Lustre: Skipped 1 previous similar message [ 9486.889692] Lustre: Skipped 1 previous similar message [ 9491.942833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9491.952126] Lustre: Skipped 1 previous similar message [ 9496.033020] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9496.361512] Lustre: server umount lustre-MDT0000 complete [ 9497.058035] LustreError: 306213:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9497.080188] LustreError: 306213:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 9500.203374] LustreError: 307007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790782370 with bad export cookie 13809088736183501153 [ 9500.215297] LustreError: 307007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9500.219083] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9500.672138] Lustre: server umount lustre-MDT0001 complete [ 9510.055907] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:32:58 (1790782378) [ 9511.549427] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9513.269489] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:33:02 (1790782382) [ 9516.952795] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 9548.753195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9549.049291] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9549.075884] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9549.342275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9555.004207] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9563.874438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9563.933513] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9563.960695] Lustre: 312033:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9570.055352] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9592.940083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9593.178590] Lustre: Denying initial registration attempt from nid 192.168.202.145@tcp, specified as failover [ 9593.188822] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9593.206518] LustreError: 313483:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 9593.219863] LustreError: 313483:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9593.227532] LustreError: 313483:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9593.329368] Lustre: server umount lustre-OST0000 complete [ 9593.336394] LustreError: 313483:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9599.970715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9599.981246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9600.010202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9614.306249] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9614.480916] Lustre: server umount lustre-MDT0000 complete [ 9617.894021] LustreError: 312043:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9617.924565] LustreError: 312043:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9618.402589] LustreError: 312023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790782489 with bad export cookie 13809088736183502035 [ 9618.410996] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9618.418791] LustreError: 312023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9618.855907] Lustre: server umount lustre-MDT0001 complete [ 9629.663887] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 9632.457122] Key type lgssc unregistered [ 9632.731620] LNet: 314603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9632.747611] LNetError: 314603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9633.768403] LNet: Removed LNI 192.168.202.145@tcp [ 9634.858942] Key type .llcrypt unregistered [ 9634.861885] Key type ._llcrypt unregistered [ 9647.877204] Key type ._llcrypt registered [ 9647.883555] Key type .llcrypt registered [ 9647.975936] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 9649.275822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9649.413779] alg: No test for adler32 (adler32-zlib) [ 9650.587894] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 9651.095624] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 9652.832200] Key type lgssc registered [ 9654.291648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9704.299500] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 9713.256519] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9713.292815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9714.588642] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9714.621665] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9714.761088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9719.626299] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9727.968315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9728.095908] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9728.139466] Lustre: 317398:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9728.392629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9733.077263] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9740.350751] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9745.849520] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9765.889128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9766.105234] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9766.440489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9773.213324] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9782.530539] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9785.828880] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9785.838445] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9785.860246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9786.346270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9786.349963] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9786.376693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9786.398102] Lustre: Skipped 1 previous similar message [ 9790.937448] Lustre: server umount lustre-OST0000 complete [ 9796.580171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9796.607865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9796.616263] Lustre: Skipped 1 previous similar message [ 9801.776200] Lustre: server umount lustre-MDT0000 complete [ 9806.303387] LustreError: 317390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790782677 with bad export cookie 16496742269427849019 [ 9806.320110] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9806.916988] Lustre: server umount lustre-MDT0001 complete [ 9817.866597] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 9820.858544] Key type lgssc unregistered [ 9821.118765] LNet: 320968:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9821.124110] LNetError: 320968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9821.141969] LNet: Removed LNI 192.168.202.145@tcp [ 9822.329153] Key type .llcrypt unregistered [ 9822.334427] Key type ._llcrypt unregistered [ 9849.088906] Key type ._llcrypt registered [ 9849.092132] Key type .llcrypt registered [ 9849.257413] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 9850.488943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9850.673383] alg: No test for adler32 (adler32-zlib) [ 9851.752548] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 9852.079966] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 9853.759164] Key type lgssc registered [ 9855.050738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9863.978366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9864.001640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9865.600934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9870.113290] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9875.940985] LustreError: 322400:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9875.968311] LustreError: 322400:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9878.838471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9879.122136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9883.214267] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9890.550678] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9896.446346] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9905.458133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9906.060352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9912.472981] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 9919.591411] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9935.330947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9935.346577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9935.382967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9936.731706] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.45@tcp (stopping) [ 9938.931134] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9938.931174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9938.957385] Lustre: Skipped 1 previous similar message [ 9938.973088] Lustre: Skipped 1 previous similar message [ 9940.804385] Lustre: server umount lustre-MDT0000 complete [ 9941.797077] LustreError: 324894:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9944.035155] LustreError: 322395:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9944.062746] LustreError: 322395:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9944.903515] LustreError: 322382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790782815 with bad export cookie 2426509711814176121 [ 9944.909281] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9944.913513] LustreError: 322382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9946.924058] LustreError: 322396:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9946.943046] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.45@tcp (stopping) [ 9946.955495] Lustre: Skipped 1 previous similar message [ 9949.153845] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9952.041711] LustreError: 324894:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9952.041873] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.45@tcp (stopping) [ 9952.073558] LustreError: 324894:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9952.088451] Lustre: Skipped 1 previous similar message [ 9959.395423] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9959.641210] Lustre: server umount lustre-MDT0001 complete [ 9965.800773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9974.782884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9990.118582] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x21acb1ba08b84c35 [ 9990.129883] Lustre: MGC192.168.202.145@tcp: Connection restored to 0@lo (at 0@lo) [ 9990.468198] LustreError: 324233:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9990.488791] LustreError: 324233:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9990.535329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9990.570758] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9994.878694] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10003.271193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10003.581213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10007.396052] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10015.224568] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10015.504533] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10017.274795] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10023.181743] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10025.129843] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10030.565221] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10030.577205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10030.585428] Lustre: Skipped 1 previous similar message [10030.603617] Lustre: Skipped 4 previous similar messages [10036.593929] Lustre: server umount lustre-OST0000 complete [10040.802618] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10047.140465] Lustre: server umount lustre-MDT0000 complete [10050.700951] LustreError: 322382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790782921 with bad export cookie 2426509711814183989 [10050.700951] LustreError: 322380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790782921 with bad export cookie 2426509711814183989 [10050.701215] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10050.706704] LustreError: 322382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10051.162785] Lustre: server umount lustre-MDT0001 complete [10063.121732] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [10066.086863] Key type lgssc unregistered [10066.415517] LNet: 328695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10066.435128] LNetError: 328695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10066.458733] LNet: Removed LNI 192.168.202.145@tcp [10067.415973] Key type .llcrypt unregistered [10067.418557] Key type ._llcrypt unregistered [10093.499263] Key type ._llcrypt registered [10093.501487] Key type .llcrypt registered [10093.577930] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [10094.748316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10094.882857] alg: No test for adler32 (adler32-zlib) [10096.058932] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [10096.327071] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [10097.992304] Key type lgssc registered [10099.244713] Lustre: Echo OBD driver; http://www.lustre.org/ [10108.244070] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10108.284606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10109.848797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10115.475223] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10120.164850] LustreError: 330127:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10120.180622] LustreError: 330127:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10124.677821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10125.114484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10129.179051] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10135.718293] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10141.241627] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10145.761530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10145.776300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10145.810287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10149.858161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10149.860267] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10150.950650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10150.964008] Lustre: Skipped 2 previous similar messages [10151.563811] Lustre: server umount lustre-MDT0000 complete [10155.795301] LustreError: 330109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790783026 with bad export cookie 8781900951757495302 [10155.818394] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10155.821934] LustreError: 330109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10156.533911] Lustre: server umount lustre-MDT0001 complete [10170.227107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10170.665610] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10170.720051] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10171.065468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10175.222935] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10184.379522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10184.476458] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10184.517921] Lustre: 332713:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10184.553944] LustreError: 332724:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10184.783894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10190.119611] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10198.889631] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10205.111793] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10215.662330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10215.988108] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10216.007076] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10216.016336] LustreError: 334597:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [10216.029351] LustreError: 334597:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [10216.040934] LustreError: 334597:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10216.185209] Lustre: server umount lustre-OST0000 complete [10216.187421] LustreError: 334597:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10228.216548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10228.616189] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10228.894065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10234.157129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10235.972513] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10244.200239] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10251.531596] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10251.651501] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10251.696631] Lustre: lustre-OST0001: new disk, initializing [10251.702545] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10251.713496] Lustre: 336141:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10251.860632] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10257.384810] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10261.090436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10261.111699] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10261.213792] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10264.327936] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10271.716649] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10271.717611] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10271.732864] LustreError: Skipped 1 previous similar message [10271.738978] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10271.784636] Lustre: Skipped 1 previous similar message [10273.059792] Lustre: server umount lustre-OST0001 complete [10276.839925] LustreError: 335068:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10276.855239] LustreError: 335068:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10280.417048] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10280.424046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10280.428730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10280.441539] Lustre: Skipped 1 previous similar message [10280.462318] Lustre: Skipped 2 previous similar messages [10281.969449] LustreError: 335067:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10281.983976] LustreError: 335067:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10283.607992] Lustre: server umount lustre-OST0000 complete [10287.098935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10287.104634] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10287.111243] Lustre: Skipped 1 previous similar message [10287.144189] LustreError: Skipped 1 previous similar message [10292.195481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10292.205825] Lustre: Skipped 5 previous similar messages [10293.778292] Lustre: server umount lustre-MDT0000 complete [10297.332176] LustreError: 332725:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10297.350342] LustreError: 332725:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10297.914422] LustreError: 336139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790783168 with bad export cookie 8781900951757495848 [10297.916240] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10297.940765] LustreError: 336139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10298.336282] Lustre: server umount lustre-MDT0001 complete [10310.938127] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [10314.141429] Key type lgssc unregistered [10314.492453] LNet: 338099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10314.502364] LNetError: 338099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10314.521760] LNet: Removed LNI 192.168.202.145@tcp [10315.559689] Key type .llcrypt unregistered [10315.563053] Key type ._llcrypt unregistered [10391.987301] Key type ._llcrypt registered [10391.991650] Key type .llcrypt registered [10392.143118] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [10411.663804] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [10413.240263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10413.707421] alg: No test for adler32 (adler32-zlib) [10415.096722] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [10415.337332] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [10417.016320] Key type lgssc registered [10418.451706] Lustre: Echo OBD driver; http://www.lustre.org/ [10457.919243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10457.956578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10459.313942] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10459.337865] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10459.411960] Lustre: lustre-MDT0000: new disk, initializing [10459.538264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10459.558334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10464.423619] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10475.541494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10475.724535] Lustre: 343054:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10475.781876] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10475.791463] Lustre: Skipped 1 previous similar message [10475.898499] Lustre: lustre-MDT0001: new disk, initializing [10476.037333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10476.113431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10476.134782] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10481.289300] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10491.026197] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10497.632593] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10505.945647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10506.316703] Lustre: lustre-OST0000: new disk, initializing [10506.320452] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10506.332426] Lustre: 345022:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10506.398347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10513.044155] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10515.508634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10515.520732] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10515.627370] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10523.784460] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10528.579930] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10528.926320] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10534.555401] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10534.790858] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10541.539242] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10541.545920] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10541.548759] LustreError: Skipped 1 previous similar message [10541.563877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10541.566024] Lustre: Skipped 1 previous similar message [10541.580451] Lustre: Skipped 1 previous similar message [10542.889489] Lustre: server umount lustre-OST0000 complete [10547.681983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10547.687886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10547.697332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10551.777651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10551.780247] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10556.902580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10556.907075] Lustre: Skipped 2 previous similar messages [10560.992968] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10561.263507] Lustre: server umount lustre-MDT0000 complete [10564.673119] LustreError: 343045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790783435 with bad export cookie 13492936694317955580 [10564.681767] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10564.684121] LustreError: 343045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10565.079179] Lustre: server umount lustre-MDT0001 complete [10572.614351] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:50:41 (1790783441) [10579.962263] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:50:48 (1790783448) [10589.556538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10590.113794] LustreError: 347777:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10590.148699] LustreError: 347777:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10590.366472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10595.810448] LustreError: 347778:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10597.488221] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10599.915427] LustreError: 347777:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10605.052957] LustreError: 347778:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10609.454536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10610.143737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10614.729074] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10622.843253] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10628.068242] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10635.010097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10635.335112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10641.903263] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10649.688773] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10660.331035] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10661.792656] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10663.560336] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10664.924450] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10667.027038] Lustre: Failing over lustre-MDT0000 [10667.339837] Lustre: server umount lustre-MDT0000 complete [10670.051806] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10670.055531] LustreError: 348507:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10670.061844] Lustre: Skipped 2 previous similar messages [10670.094523] LustreError: 348507:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10680.292060] LustreError: 347778:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10680.315837] LustreError: 347778:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [10686.435673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10686.496249] Lustre: 341267:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790783540/real 1790783540] req@000000005d8eed23 x1877772374975232/t0(0) o400->MGC192.168.202.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790783556 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10686.542918] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10697.187017] LustreError: 349678:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10697.199535] LustreError: 341264:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c5d9b31c x1877772374981376/t0(0) o250->MGC192.168.202.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10697.214678] LustreError: 349678:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 14 previous similar messages [10697.507492] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.45@tcp (not set up) [10697.755142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10697.870240] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10700.728222] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10702.884369] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10702.932107] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10703.020295] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10705.230877] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10717.260837] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10719.504317] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10723.524351] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10725.707170] Lustre: Failing over lustre-MDT0000 [10726.174812] Lustre: server umount lustre-MDT0000 complete [10728.417903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10728.441360] Lustre: Skipped 1 previous similar message [10731.295831] LustreError: 348789:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10731.306637] LustreError: 348789:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10744.800122] Lustre: 341266:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790783599/real 1790783599] req@00000000ee44d516 x1877772374998528/t0(0) o400->MGC192.168.202.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790783615 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10744.826791] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10745.404685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10755.443287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10755.525119] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10757.927000] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10760.187669] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10760.692396] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10760.701411] Lustre: Skipped 2 previous similar messages [10760.744984] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10760.814544] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10769.239866] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10770.503553] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10781.167505] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10781.184997] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10781.203322] Lustre: Skipped 3 previous similar messages [10781.203704] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10781.236910] Lustre: Skipped 2 previous similar messages [10783.726224] Lustre: server umount lustre-MDT0000 complete [10789.391915] LustreError: 347757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790783660 with bad export cookie 13492936694317958303 [10789.416448] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10789.424382] LustreError: 347757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10790.250276] Lustre: server umount lustre-MDT0001 complete [10805.970820] Lustre: server umount lustre-OST0000 complete [10823.030057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10823.447299] LustreError: 354126:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10823.463423] LustreError: 354126:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 26 previous similar messages [10823.506075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10826.630099] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10835.389319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10840.753743] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10850.219653] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10856.948695] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10864.370455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10868.862457] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10871.653910] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10879.291477] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10889.707076] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10889.720937] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10889.746576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10895.480643] Lustre: server umount lustre-OST0000 complete [10905.723522] Lustre: server umount lustre-MDT0000 complete [10909.867550] LustreError: 354107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790783780 with bad export cookie 13492936694317959045 [10909.881477] LustreError: 354107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10909.883344] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10910.147497] Lustre: server umount lustre-MDT0001 complete [10920.000859] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [10922.812756] Key type lgssc unregistered [10923.106558] LNet: 357656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10923.126647] LNetError: 357656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10923.156393] LNet: Removed LNI 192.168.202.145@tcp [10924.114509] Key type .llcrypt unregistered [10924.118802] Key type ._llcrypt unregistered [10950.646563] Key type ._llcrypt registered [10950.648803] Key type .llcrypt registered [10950.763994] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [10952.078901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10952.149784] alg: No test for adler32 (adler32-zlib) [10953.362796] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [10953.758125] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [10955.432208] Key type lgssc registered [10956.651502] Lustre: Echo OBD driver; http://www.lustre.org/ [10964.265961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10964.281606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10965.829467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10970.034733] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10976.229615] LustreError: 359188:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10976.247103] LustreError: 359188:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10978.378045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10979.003215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10984.365645] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [10994.581281] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11001.265680] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11008.052473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11008.273470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11010.363551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11015.041651] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11024.033289] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11039.205477] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11039.206653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11039.224050] Lustre: Skipped 2 previous similar messages [11039.241474] Lustre: Skipped 2 previous similar messages [11043.248210] Lustre: server umount lustre-MDT0000 complete [11044.323532] LustreError: 359916:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11044.357273] LustreError: 359916:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11047.189352] LustreError: 359923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790783917 with bad export cookie 9165528268002389719 [11047.193755] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11047.204474] LustreError: 359923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11047.777700] Lustre: server umount lustre-MDT0001 complete [11062.161696] Lustre: server umount lustre-OST0000 complete [11076.048748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11076.477615] LustreError: 362920:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11076.562655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11081.511925] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11081.699439] LustreError: 362921:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11086.829562] LustreError: 362920:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11092.088728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11092.852747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11099.140312] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11107.647778] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11113.959986] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11120.567751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11120.961243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11121.986161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11128.203396] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11135.720743] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11149.281637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11149.299053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11149.315888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11152.866165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11152.868153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11152.887600] Lustre: Skipped 1 previous similar message [11152.923347] Lustre: Skipped 1 previous similar message [11154.345100] Lustre: server umount lustre-MDT0000 complete [11157.994695] LustreError: 362917:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11158.029209] LustreError: 362917:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11158.437532] LustreError: 362900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790784029 with bad export cookie 9165528268002390832 [11158.441351] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11158.444976] LustreError: 362900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11158.805850] Lustre: server umount lustre-MDT0001 complete [11173.265453] Lustre: server umount lustre-OST0000 complete [11188.413686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11188.793994] LustreError: 366830:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11188.851505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11193.680557] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11203.158036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11203.555715] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11203.559106] Lustre: Skipped 1 previous similar message [11210.183259] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11220.207491] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11226.894301] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11234.568167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11235.076558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11235.086119] Lustre: Skipped 1 previous similar message [11241.906235] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11250.626907] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11258.614277] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11269.102721] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11269.105428] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11269.110518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11269.131471] LustreError: Skipped 1 previous similar message [11269.159564] Lustre: Skipped 1 previous similar message [11272.846351] Lustre: server umount lustre-OST0000 complete [11279.329621] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11279.336113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11279.341212] Lustre: Skipped 1 previous similar message [11279.344392] Lustre: Skipped 1 previous similar message [11282.999496] Lustre: server umount lustre-MDT0000 complete [11284.455562] LustreError: 367772:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11284.471949] LustreError: 367772:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11286.666076] LustreError: 367577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790784157 with bad export cookie 9165528268002391847 [11286.678856] LustreError: 367577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11286.682747] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11287.078473] Lustre: server umount lustre-MDT0001 complete [11299.499208] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [11302.410383] Key type lgssc unregistered [11302.753870] LNet: 370397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11302.758201] LNetError: 370397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11302.769987] LNet: Removed LNI 192.168.202.145@tcp [11303.620572] Key type .llcrypt unregistered [11303.625796] Key type ._llcrypt unregistered [11329.593399] Key type ._llcrypt registered [11329.595790] Key type .llcrypt registered [11329.748760] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [11331.211842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11331.221152] alg: No test for adler32 (adler32-zlib) [11332.526830] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [11333.040625] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [11335.016889] Key type lgssc registered [11337.445083] Lustre: Echo OBD driver; http://www.lustre.org/ [11346.866468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11346.881927] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11346.888457] LustreError: 371857:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11346.903202] LustreError: 371857:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11346.909545] LustreError: 371857:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11346.914889] LustreError: 371857:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11346.920929] LustreError: 371857:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11356.558679] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11356.567876] LustreError: 372254:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11356.589810] LustreError: 372254:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11356.602718] LustreError: 372254:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11356.621656] LustreError: 372254:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11356.631966] LustreError: 372254:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11383.317645] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [11388.255571] Key type lgssc unregistered [11388.743466] LNet: 373257:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11388.751949] LNetError: 373257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11388.781482] LNet: Removed LNI 192.168.202.145@tcp [11390.050089] Key type .llcrypt unregistered [11390.069819] Key type ._llcrypt unregistered [11413.528663] Key type ._llcrypt registered [11413.532388] Key type .llcrypt registered [11413.677623] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [11434.864151] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [11437.431496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11437.618495] alg: No test for adler32 (adler32-zlib) [11438.835815] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [11439.075704] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [11440.736209] Key type lgssc registered [11442.524815] Lustre: Echo OBD driver; http://www.lustre.org/ [11485.004214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11485.027447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11486.230782] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11486.258874] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11486.304593] Lustre: lustre-MDT0000: new disk, initializing [11486.394283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11486.411628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11491.678360] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11506.310522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11506.562273] Lustre: 376430:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11506.699969] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11506.712765] Lustre: Skipped 1 previous similar message [11506.896858] Lustre: lustre-MDT0001: new disk, initializing [11507.113895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11507.204408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11507.224196] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11515.349224] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11526.573646] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11533.482805] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11542.560674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11542.981458] Lustre: lustre-OST0000: new disk, initializing [11542.992657] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11543.007834] Lustre: 378399:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11543.169851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11550.030610] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11552.331278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11552.341792] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11552.515701] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11560.427933] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11566.240613] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11566.517385] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11572.095523] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11572.348869] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11578.341634] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11578.343180] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11578.355967] LustreError: Skipped 1 previous similar message [11578.362390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11578.377927] Lustre: Skipped 1 previous similar message [11578.407373] Lustre: Skipped 1 previous similar message [11580.334720] Lustre: server umount lustre-OST0000 complete [11588.586181] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11588.599852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11588.608781] Lustre: Skipped 1 previous similar message [11593.698018] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11593.704111] Lustre: Skipped 1 previous similar message [11598.818435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11598.832870] Lustre: Skipped 1 previous similar message [11599.328913] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11599.830455] Lustre: server umount lustre-MDT0000 complete [11604.637643] LustreError: 379940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790784475 with bad export cookie 16653260148060384396 [11604.646704] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11604.657133] LustreError: 379940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11605.245831] Lustre: server umount lustre-MDT0001 complete [11616.606553] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11618.610615] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 12:08:07 (1790784487) [11626.594767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11627.216226] LustreError: 380862:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11627.242707] LustreError: 380862:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11627.347498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11631.737470] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11632.611179] LustreError: 380863:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11636.710832] LustreError: 380862:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11640.549544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11640.825437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11645.710810] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11653.799273] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11660.496824] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11668.528499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11669.176221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11676.950083] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11686.985154] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11695.609508] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11695.860555] Lustre: lustre-OST0001: new disk, initializing [11695.864245] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11695.868806] Lustre: 383700:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11695.952961] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11701.874369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11701.886440] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11702.098924] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11704.066507] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11714.836267] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11722.043680] Lustre: lustre-OST0001: Not available for connect from 192.168.202.45@tcp (stopping) [11722.740633] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11722.773395] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11722.777936] Lustre: Skipped 1 previous similar message [11722.806668] LustreError: Skipped 1 previous similar message [11724.770054] Lustre: server umount lustre-OST0001 complete [11727.138814] LustreError: 384176:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11733.016127] LustreError: 382687:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11733.050550] LustreError: 382687:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11743.236628] LustreError: 382689:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11743.262080] LustreError: 382689:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11744.225252] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11744.228084] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11744.259477] Lustre: Skipped 1 previous similar message [11744.266382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11744.268834] Lustre: Skipped 2 previous similar messages [11747.302696] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11749.885763] Lustre: server umount lustre-OST0000 complete [11758.565212] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11758.570352] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11758.589312] Lustre: Skipped 2 previous similar messages [11761.241812] Lustre: server umount lustre-MDT0000 complete [11763.687805] LustreError: 380862:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11763.707199] LustreError: 380862:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11765.897529] LustreError: 380844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790784636 with bad export cookie 16653260148060385278 [11765.898508] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11765.907967] LustreError: 380844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11772.486195] Lustre: server umount lustre-MDT0001 complete [11784.221720] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [11787.482821] Key type lgssc unregistered [11787.886297] LNet: 385687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11787.900413] LNetError: 385687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11787.936343] LNet: Removed LNI 192.168.202.145@tcp [11789.140935] Key type .llcrypt unregistered [11789.142590] Key type ._llcrypt unregistered [11845.151658] Key type ._llcrypt registered [11845.154180] Key type .llcrypt registered [11845.301987] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [11846.312295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11846.351401] alg: No test for adler32 (adler32-zlib) [11847.691715] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [11848.042823] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [11849.776236] Key type lgssc registered [11851.261977] Lustre: Echo OBD driver; http://www.lustre.org/ [11859.891721] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11859.939317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11861.475285] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11861.507399] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11861.837629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11867.631687] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11878.415857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11878.553026] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11878.588523] Lustre: 388502:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11878.932054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11884.268570] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11893.554769] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11899.839479] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11903.459362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11903.468042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11903.494524] Lustre: Skipped 1 previous similar message [11903.511321] Lustre: Skipped 1 previous similar message [11908.485976] Lustre: server umount lustre-MDT0000 complete [11912.703756] LustreError: 388493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790784783 with bad export cookie 9999485292387182218 [11912.705758] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11912.735032] LustreError: 388493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11913.174777] Lustre: server umount lustre-MDT0001 complete [11923.893174] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 12:13:12 (1790784792) [11931.101859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11931.629560] LustreError: 391015:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11931.656014] LustreError: 391015:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11931.714922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11936.752820] LustreError: 391016:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11937.020939] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11941.870144] LustreError: 391015:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11946.280543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11946.845487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11951.778882] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11959.279550] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11966.906983] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11974.392788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11974.641707] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11974.856061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11981.641665] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [11989.768046] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11997.851819] Lustre: Permanently deactivating lustre-OST0000 [11997.870647] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12003.917975] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12005.656819] LustreError: 393956:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12005.665953] LustreError: 393956:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12005.673962] LustreError: 393956:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12009.262336] LustreError: 394005:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12009.270974] LustreError: 394005:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12017.447154] Lustre: server umount lustre-OST0000 complete [12020.331863] LustreError: 394255:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12020.338553] LustreError: 394255:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12023.779397] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12023.790913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12023.815268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12029.411911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12029.419520] Lustre: Skipped 3 previous similar messages [12031.600947] Lustre: server umount lustre-MDT0000 complete [12034.532142] LustreError: 391015:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12034.549172] LustreError: 391015:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12035.386170] LustreError: 391747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790784906 with bad export cookie 9999485292387182799 [12035.397450] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12035.397991] LustreError: 391747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12046.060166] Lustre: server umount lustre-MDT0001 complete [12055.359653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12060.358787] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [12063.383724] LustreError: 395415:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12070.183851] Lustre: 395753:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.145@tcp [12072.464757] Lustre: server umount MGS complete [12085.140938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12085.463469] LustreError: 396478:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12085.487107] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12085.491168] Lustre: Skipped 1 previous similar message [12085.547179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12090.039808] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [12095.976558] LustreError: 396479:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12096.010140] LustreError: 396479:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12098.627386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12098.977320] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12103.968180] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [12112.471911] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12118.849510] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12120.579426] Lustre: Permanently reactivating lustre-OST0000 [12120.594942] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12120.598843] Lustre: Skipped 2 previous similar messages [12149.231053] LustreError: 398276:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12157.714909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12158.059510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12158.067332] Lustre: Skipped 1 previous similar message [12164.579202] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [12173.127261] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12179.961458] Lustre: server umount lustre-OST0000 complete [12186.081061] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12186.087904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12186.098047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12186.109532] Lustre: Skipped 2 previous similar messages [12191.203654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12191.215537] Lustre: Skipped 2 previous similar messages [12194.369695] Lustre: server umount lustre-MDT0000 complete [12195.299726] Lustre: 387816:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790785050/real 1790785050] req@0000000015b24b85 x1877773876560512/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1790785066 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12195.350320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12195.374922] LustreError: 396478:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12195.401406] LustreError: 396478:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12196.322223] Lustre: 387814:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790785051/real 1790785051] req@00000000176abd74 x1877773876561280/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790785067 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12198.004382] LustreError: 396458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790785068 with bad export cookie 9999485292387183849 [12198.005098] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12198.011475] LustreError: 396458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12198.385743] Lustre: server umount lustre-MDT0001 complete [12211.315958] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [12214.372919] Key type lgssc unregistered [12214.841269] LNet: 400841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12214.847534] LNetError: 400841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12214.888433] LNet: Removed LNI 192.168.202.145@tcp [12216.189091] Key type .llcrypt unregistered [12216.191805] Key type ._llcrypt unregistered [12237.858778] Key type ._llcrypt registered [12237.860390] Key type .llcrypt registered [12238.023534] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [12256.149694] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [12257.522072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12257.556463] alg: No test for adler32 (adler32-zlib) [12258.644803] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [12258.869124] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [12260.544268] Key type lgssc registered [12261.905816] Lustre: Echo OBD driver; http://www.lustre.org/ [12298.715515] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12298.775159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12300.161738] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12300.240097] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12300.412202] Lustre: lustre-MDT0000: new disk, initializing [12300.497758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12300.512197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12306.069683] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [12317.550854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12317.662167] Lustre: 404021:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12317.725641] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12317.733555] Lustre: Skipped 1 previous similar message [12317.790786] Lustre: lustre-MDT0001: new disk, initializing [12317.903849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12317.945146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12317.967091] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12323.170534] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [12332.873699] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12339.240457] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12346.026242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12346.314785] Lustre: lustre-OST0000: new disk, initializing [12346.317780] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12346.328739] Lustre: 405988:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12346.489598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12353.741593] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [12356.206417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12356.213767] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12356.337543] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12364.198943] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12369.554917] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12369.765463] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12373.879411] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12374.108575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12377.063597] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12377.068176] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12377.074838] LustreError: Skipped 1 previous similar message [12377.076957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12377.108849] Lustre: Skipped 1 previous similar message [12382.178570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12382.185574] Lustre: Skipped 2 previous similar messages [12382.806889] Lustre: server umount lustre-OST0000 complete [12387.310992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12387.342368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12387.344688] Lustre: Skipped 1 previous similar message [12387.369832] Lustre: Skipped 1 previous similar message [12392.418044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12392.431281] Lustre: Skipped 1 previous similar message [12393.642392] Lustre: server umount lustre-MDT0000 complete [12398.863292] LustreError: 404011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790785269 with bad export cookie 1657556876647728746 [12398.874818] LustreError: 404011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12398.875131] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12399.202401] Lustre: server umount lustre-MDT0001 complete [12409.195987] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12411.571368] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 12:21:19 (1790785279) [12419.920711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12420.610626] LustreError: 408448:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12420.633521] LustreError: 408448:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12420.696884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12424.738581] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [12425.702661] LustreError: 408449:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12430.826420] LustreError: 408448:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12434.704435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12435.064266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12440.345732] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [12448.552406] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12454.126924] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12461.632613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12462.169061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12469.872476] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [12479.242220] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12481.286242] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12483.978183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12490.213426] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12490.237264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12490.242995] Lustre: Skipped 1 previous similar message [12490.280398] Lustre: Skipped 1 previous similar message [12495.615201] Lustre: server umount lustre-OST0000 complete [12500.451033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12500.457839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12500.478583] Lustre: Skipped 1 previous similar message [12500.486086] Lustre: Skipped 2 previous similar messages [12505.569524] LustreError: 408449:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12505.577048] Lustre: server umount lustre-MDT0000 complete [12505.595300] LustreError: 408449:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12509.548363] LustreError: 408429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790785380 with bad export cookie 1657556876647729628 [12509.562280] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12509.562757] LustreError: 408429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12510.052691] Lustre: server umount lustre-MDT0001 complete [12523.224841] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [12526.471207] Key type lgssc unregistered [12526.787743] LNet: 412173:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12526.804121] LNetError: 412173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12527.854669] LNet: Removed LNI 192.168.202.145@tcp [12528.974147] Key type .llcrypt unregistered [12528.977772] Key type ._llcrypt unregistered [12559.155869] Key type ._llcrypt registered [12559.158628] Key type .llcrypt registered [12559.265676] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [12559.544813] Key type .llcrypt unregistered [12559.549058] Key type ._llcrypt unregistered [12578.495734] Key type ._llcrypt registered [12578.504320] Key type .llcrypt registered [12578.739347] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [12580.454980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12580.620365] alg: No test for adler32 (adler32-zlib) [12581.723658] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [12582.128623] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [12583.920303] Key type lgssc registered [12585.658821] Lustre: Echo OBD driver; http://www.lustre.org/ [12594.588660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12594.628391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12596.283923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12602.619450] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [12605.413394] LustreError: 414407:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12605.436780] LustreError: 414407:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12610.550672] LustreError: 414408:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12613.030146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12613.321643] LustreError: 414407:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12613.345430] LustreError: 414407:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12613.364686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12618.724588] LustreError: 414408:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12618.749846] LustreError: 414408:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12620.785849] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12621.100534] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [12625.896454] LustreError: 414402:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12625.906127] LustreError: 414402:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12628.969538] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12636.145519] LustreError: 414407:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12636.181864] LustreError: 414407:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12637.185617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12637.513515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12642.399877] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [12651.492295] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12651.498144] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12651.508867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12653.026026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12653.028677] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12653.030904] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12653.035310] Lustre: Skipped 1 previous similar message [12654.017329] Lustre: server umount lustre-OST0000 complete [12663.267903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12663.268123] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12663.283657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12663.289040] Lustre: Skipped 1 previous similar message [12665.570112] Lustre: server umount lustre-MDT0000 complete [12668.397232] LustreError: 414402:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12668.429083] LustreError: 414402:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12669.983515] LustreError: 414387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790785540 with bad export cookie 10650440566651391978 [12670.008532] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12670.536692] Lustre: server umount lustre-MDT0001 complete [12685.044710] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [12688.719683] Key type lgssc unregistered [12689.136852] LNet: 417556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12689.145563] LNetError: 417556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12689.179847] LNet: Removed LNI 192.168.202.145@tcp [12690.459743] Key type .llcrypt unregistered [12690.466079] Key type ._llcrypt unregistered [12719.985450] Key type ._llcrypt registered [12719.986752] Key type .llcrypt registered [12720.148966] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [12721.357109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12721.489990] alg: No test for adler32 (adler32-zlib) [12722.646505] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [12722.860872] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [12724.520230] Key type lgssc registered [12726.195562] Lustre: Echo OBD driver; http://www.lustre.org/ [12735.969890] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12736.028856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12751.648523] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12756.770337] LustreError: 418965:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.145@tcp: failed processing log, type 4: rc = -110 [12783.456333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12790.241284] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [12797.794616] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12804.239049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12804.722199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12804.817666] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12810.215722] LustreError: 418990:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12810.235894] LustreError: 418990:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12810.935422] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [12815.341144] LustreError: 418991:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12817.385544] LustreError: 418990:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12817.409397] LustreError: 418990:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12820.675678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12821.062198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12827.800307] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [12836.082415] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12842.440809] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12850.661433] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12850.670424] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12850.739255] LustreError: 418980:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12850.824280] Lustre: server umount lustre-OST0000 complete [12856.805946] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12856.806177] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12856.826140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12856.852742] LustreError: Skipped 1 previous similar message [12857.670188] Lustre: server umount lustre-MDT0000 complete [12861.822494] LustreError: 418973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790785732 with bad export cookie 9431452047401258145 [12861.828503] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12861.840132] LustreError: 418973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12872.333231] Lustre: server umount lustre-MDT0001 complete [12888.050534] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [12891.575748] Key type lgssc unregistered [12891.930641] LNet: 422533:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12891.942823] LNetError: 422533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12892.969841] LNet: Removed LNI 192.168.202.145@tcp [12893.669350] Key type .llcrypt unregistered [12893.672425] Key type ._llcrypt unregistered [12922.422234] Key type ._llcrypt registered [12922.424179] Key type .llcrypt registered [12922.559233] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [12924.344711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12924.392548] alg: No test for adler32 (adler32-zlib) [12925.505123] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [12925.714770] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [12927.368254] Key type lgssc registered [12928.887225] Lustre: Echo OBD driver; http://www.lustre.org/ [12937.071167] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12937.097411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12938.635194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12944.073343] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [12948.963470] LustreError: 423967:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12949.013256] LustreError: 423967:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12954.082487] LustreError: 423968:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12954.100852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12954.453410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12960.503200] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [12970.216706] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12977.117656] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12984.498152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12984.796453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12992.149970] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [13001.402739] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13008.365780] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13016.055278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13016.063472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13016.091383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13018.433156] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.45@tcp (stopping) [13018.596043] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13018.603679] Lustre: Skipped 1 previous similar message [13022.418287] Lustre: server umount lustre-MDT0000 complete [13023.525713] LustreError: 423962:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13025.046717] LustreError: 423948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.45@tcp arrived at 1790785895 with bad export cookie 4607326343372064572 [13025.069867] LustreError: 423948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13026.990976] LustreError: 426686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790785897 with bad export cookie 4607326343372063907 [13027.003150] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13028.834069] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13028.837707] LustreError: 424695:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13028.862799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13028.895137] LustreError: 424695:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [13028.916197] Lustre: Skipped 3 previous similar messages [13031.844747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13033.335518] Lustre: server umount lustre-OST0000 complete [13036.963185] LustreError: 424695:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13036.991474] LustreError: 424695:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [13041.383184] Lustre: server umount lustre-MDT0001 complete [13053.595302] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [13056.506333] Key type lgssc unregistered [13056.867672] LNet: 427675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13056.872843] LNetError: 427675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13056.890340] LNet: Removed LNI 192.168.202.145@tcp [13057.941490] Key type .llcrypt unregistered [13057.946094] Key type ._llcrypt unregistered [13083.626562] Key type ._llcrypt registered [13083.628890] Key type .llcrypt registered [13083.718771] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [13085.268363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13085.305300] alg: No test for adler32 (adler32-zlib) [13086.380704] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [13086.594035] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [13088.248494] Key type lgssc registered [13089.141062] Lustre: Echo OBD driver; http://www.lustre.org/ [13096.407068] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [13096.444393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13098.077366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13102.522846] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [13108.195414] LustreError: 429108:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13108.242027] LustreError: 429108:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13113.316830] LustreError: 429109:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13114.687352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13115.048489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13120.653546] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [13128.515482] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13133.597548] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13139.220240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13139.727980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13142.902794] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13145.867935] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [13156.708707] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13163.492392] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13163.507180] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13163.507243] Lustre: Skipped 1 previous similar message [13163.523382] Lustre: Skipped 1 previous similar message [13164.870437] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.45@tcp (stopping) [13168.127765] Lustre: server umount lustre-MDT0001 complete [13168.617455] LustreError: 429108:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13168.642394] LustreError: 429108:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13173.731525] LustreError: 429109:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13173.767769] LustreError: 429109:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13178.856060] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13178.857944] LustreError: 429836:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13178.865716] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13178.871811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13181.994058] Lustre: server umount lustre-OST0000 complete [13186.298700] Lustre: server umount lustre-MDT0000 complete [13202.317641] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [13205.692196] Key type lgssc unregistered [13206.052481] LNet: 432794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13206.064635] LNetError: 432794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13206.102587] LNet: Removed LNI 192.168.202.145@tcp [13207.234368] Key type .llcrypt unregistered [13207.236415] Key type ._llcrypt unregistered [13234.496433] Key type ._llcrypt registered [13234.500848] Key type .llcrypt registered [13234.613241] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [13250.071346] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [13251.272159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13251.441867] alg: No test for adler32 (adler32-zlib) [13252.591756] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [13252.927600] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [13254.656232] Key type lgssc registered [13256.110252] Lustre: Echo OBD driver; http://www.lustre.org/ [13293.425472] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [13309.288228] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [13309.322111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13310.701956] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13310.739812] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13310.858750] Lustre: lustre-MDT0000: new disk, initializing [13310.965860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13310.990557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13315.891573] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [13331.726961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13332.005527] Lustre: lustre-OST0000: new disk, initializing [13332.011915] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13332.014460] Lustre: Skipped 1 previous similar message [13332.018609] Lustre: 437843:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13332.103844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13338.534856] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [13340.271272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13340.274856] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13340.287161] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13348.140750] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13363.351613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13363.500503] Lustre: 436825:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13363.603297] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13363.749263] Lustre: lustre-MDT0001: new disk, initializing [13363.922751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13364.017977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13364.029493] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13369.403202] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [13378.801359] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13378.964771] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13383.671790] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13383.929130] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13391.330423] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13391.331824] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13391.350775] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13391.367471] Lustre: Skipped 1 previous similar message [13396.472888] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13396.491277] Lustre: Skipped 2 previous similar messages [13401.573933] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13401.590887] Lustre: Skipped 2 previous similar messages [13403.104146] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13403.341541] Lustre: server umount lustre-MDT0001 complete [13411.817634] LustreError: 436831:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13411.845983] LustreError: 436831:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [13416.940334] LustreError: 436832:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13416.958042] LustreError: 436832:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13422.050955] LustreError: 436837:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13422.053085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13422.099431] Lustre: Skipped 1 previous similar message [13422.104779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13422.113516] Lustre: Skipped 1 previous similar message [13426.875483] Lustre: server umount lustre-OST0000 complete [13427.178281] LustreError: 436836:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13437.367083] Lustre: server umount lustre-MDT0000 complete [13456.078310] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [13459.412967] Key type lgssc unregistered [13459.757337] LNet: 441592:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13459.763531] LNetError: 441592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13459.783781] LNet: Removed LNI 192.168.202.145@tcp [13460.790342] Key type .llcrypt unregistered [13460.795720] Key type ._llcrypt unregistered [13500.452869] Key type ._llcrypt registered [13500.454994] Key type .llcrypt registered [13500.522102] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [13500.725564] Key type .llcrypt unregistered [13500.728667] Key type ._llcrypt unregistered [13516.446369] Key type ._llcrypt registered [13516.448489] Key type .llcrypt registered [13516.549682] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [13517.887650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13517.987322] alg: No test for adler32 (adler32-zlib) [13519.210683] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [13519.510747] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [13521.216246] Key type lgssc registered [13522.616593] Lustre: Echo OBD driver; http://www.lustre.org/ [13549.310986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [13549.327934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13550.478818] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13550.506023] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13550.625364] Lustre: lustre-MDT0000: new disk, initializing [13550.767481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13550.810306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13555.337976] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [13566.217762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13566.390298] Lustre: 444981:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13566.465590] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13566.468734] Lustre: Skipped 1 previous similar message [13566.528768] Lustre: lustre-MDT0001: new disk, initializing [13566.610486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13566.654319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13566.662740] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13571.122984] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [13582.158799] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13589.208784] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13597.005462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13597.335093] Lustre: lustre-OST0000: new disk, initializing [13597.339846] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13597.348592] Lustre: 446950:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13597.442524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13604.737447] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [13607.016749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13607.035507] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13607.218558] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13613.543167] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13622.758587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13622.761439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13622.780582] Lustre: Skipped 1 previous similar message [13622.805430] Lustre: Skipped 2 previous similar messages [13627.644869] Lustre: server umount lustre-MDT0000 complete [13627.884209] LustreError: 444992:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13627.903915] LustreError: 444992:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [13632.010807] LustreError: 444972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790786502 with bad export cookie 958003517629226876 [13632.017725] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13632.283836] Lustre: server umount lustre-MDT0001 complete [13646.991923] Lustre: server umount lustre-OST0000 complete [13658.506456] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:42:07 (1790786527) [13667.153182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13667.573722] LustreError: 448976:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13667.632842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13672.222650] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [13672.931206] LustreError: 448977:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13678.054849] LustreError: 448976:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13680.503700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13680.899880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13685.045690] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [13693.118212] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13699.966537] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13720.834642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13721.067888] Lustre: Found index 0 for lustre-OST0000, updating log [13721.272877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13726.799857] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [13735.268848] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13738.734542] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13749.241527] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13749.254280] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13749.264201] Lustre: Skipped 1 previous similar message [13749.268841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13750.383850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13750.397170] Lustre: Skipped 1 previous similar message [13752.988587] Lustre: server umount lustre-OST0000 complete [13772.262965] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13772.269382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13772.285293] Lustre: Skipped 2 previous similar messages [13772.313766] Lustre: Skipped 2 previous similar messages [13776.309775] Lustre: server umount lustre-MDT0000 complete [13776.418307] LustreError: 448971:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13776.449543] LustreError: 448971:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13780.716533] LustreError: 448958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790786651 with bad export cookie 958003517629228269 [13780.721142] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13780.747638] LustreError: 448958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [13781.131523] Lustre: server umount lustre-MDT0001 complete [13783.717040] LNet: 1 local NIs in recovery (showing 1): 192.168.202.145@tcp [13792.688602] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:44:21 (1790786661) [14414.370912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14414.756427] LustreError: 455395:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14414.776613] LustreError: 455395:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [14414.870415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14419.938050] LustreError: 455396:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14420.472611] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [14425.058092] LustreError: 455395:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14429.499103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14429.891256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14435.183924] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [14443.297472] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14449.271488] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14469.553532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14469.868437] Lustre: Found index 0 for lustre-OST0000, updating log [14470.088031] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14474.745953] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [14474.983254] LNet: 1 local NIs in recovery (showing 1): 192.168.202.145@tcp [14483.687550] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14488.171466] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14488.188589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14492.460946] Lustre: server umount lustre-OST0000 complete [14496.738773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14496.750037] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14496.757938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14497.768034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14497.770364] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14499.876281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14499.878226] Lustre: Skipped 3 previous similar messages [14502.952116] LustreError: 455396:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14502.964339] LustreError: 455396:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14503.028335] Lustre: server umount lustre-MDT0000 complete [14506.688079] LustreError: 455376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790787377 with bad export cookie 958003517629229354 [14506.707459] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14507.084519] Lustre: server umount lustre-MDT0001 complete [14537.458918] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [14538.464275] LNet: 1 local NIs in recovery (showing 1): 192.168.202.145@tcp [14550.639295] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [14588.709681] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:57:37 (1790787457) [14607.061162] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [14610.596737] Key type lgssc unregistered [14610.878778] LNet: 462009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14610.883764] LNetError: 462009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14730.729939] LNet: Removed LNI 192.168.202.145@tcp [14731.801651] Key type .llcrypt unregistered [14731.805588] Key type ._llcrypt unregistered [14751.905811] Key type ._llcrypt registered [14751.907687] Key type .llcrypt registered [14752.063345] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [14753.682316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14753.737270] alg: No test for adler32 (adler32-zlib) [14754.891635] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [14755.117195] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [14756.856205] Key type lgssc registered [14758.544914] Lustre: Echo OBD driver; http://www.lustre.org/ [14763.884129] LNet: Added LNI 192.168.202.145@tcp100 [8/256/0/180] [14764.750377] LNet: Added LNI 192.168.202.145@tcp101 [8/256/0/180] [14765.468228] LNet: Added LNI 192.168.202.145@tcp102 [8/256/0/180] [14766.384858] LNet: Added LNI 192.168.202.145@tcp103 [8/256/0/180] [14767.150455] LNet: Added LNI 192.168.202.145@tcp104 [8/256/0/180] [14767.867328] LNet: Added LNI 192.168.202.145@tcp105 [8/256/0/180] [14768.606480] LNet: Added LNI 192.168.202.145@tcp106 [8/256/0/180] [14769.349361] LNet: Added LNI 192.168.202.145@tcp107 [8/256/0/180] [14770.227993] LNet: Added LNI 192.168.202.145@tcp108 [8/256/0/180] [14771.174769] LNet: Added LNI 192.168.202.145@tcp109 [8/256/0/180] [14772.006640] LNet: Added LNI 192.168.202.145@tcp110 [8/256/0/180] [14773.009575] LNet: Added LNI 192.168.202.145@tcp111 [8/256/0/180] [14773.918851] LNet: Added LNI 192.168.202.145@tcp112 [8/256/0/180] [14774.711039] LNet: Added LNI 192.168.202.145@tcp113 [8/256/0/180] [14775.401660] LNet: Added LNI 192.168.202.145@tcp114 [8/256/0/180] [14776.314764] LNet: Added LNI 192.168.202.145@tcp115 [8/256/0/180] [14777.232573] LNet: Added LNI 192.168.202.145@tcp116 [8/256/0/180] [14778.195312] LNet: Added LNI 192.168.202.145@tcp117 [8/256/0/180] [14779.203624] LNet: Added LNI 192.168.202.145@tcp118 [8/256/0/180] [14780.339903] LNet: Added LNI 192.168.202.145@tcp119 [8/256/0/180] [14781.288755] LNet: Added LNI 192.168.202.145@tcp120 [8/256/0/180] [14782.219464] LNet: Added LNI 192.168.202.145@tcp121 [8/256/0/180] [14783.140432] LNet: Added LNI 192.168.202.145@tcp122 [8/256/0/180] [14784.122593] LNet: Added LNI 192.168.202.145@tcp123 [8/256/0/180] [14785.160419] LNet: Added LNI 192.168.202.145@tcp124 [8/256/0/180] [14786.217603] LNet: Added LNI 192.168.202.145@tcp125 [8/256/0/180] [14787.160716] LNet: Added LNI 192.168.202.145@tcp126 [8/256/0/180] [14788.045973] LNet: Added LNI 192.168.202.145@tcp127 [8/256/0/180] [14789.053360] LNet: Added LNI 192.168.202.145@tcp128 [8/256/0/180] [14790.053642] LNet: Added LNI 192.168.202.145@tcp129 [8/256/0/180] [14791.296659] LNet: Added LNI 192.168.202.145@tcp130 [8/256/0/180] [14792.344808] LNet: Added LNI 192.168.202.145@tcp131 [8/256/0/180] [14793.350327] LNet: Added LNI 192.168.202.145@tcp132 [8/256/0/180] [14794.356084] LNet: Added LNI 192.168.202.145@tcp133 [8/256/0/180] [14795.266742] LNet: Added LNI 192.168.202.145@tcp134 [8/256/0/180] [14796.241112] LNet: Added LNI 192.168.202.145@tcp135 [8/256/0/180] [14803.351349] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [14803.392523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14804.529784] Lustre: 465070:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.145@tcp: can't add NID 192.168.202.145@tcp130: rc = -75 [14804.597858] LNetError: 465090:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.145 [14804.613388] LNetError: Protocol error connecting to 192.168.202.145@tcp117 at host 192.168.202.145:988: is it running a compatible version of Lustre? [14804.628599] LNetError: Skipped 4 previous similar messages [14804.738798] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14804.774659] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14804.861464] Lustre: lustre-MDT0000: new disk, initializing [14804.970973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14804.992586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14805.289914] LNet: 6 local NIs in recovery (showing 6): 192.168.202.145@tcp124, 192.168.202.145@tcp125, 192.168.202.145@tcp126, 192.168.202.145@tcp127, 192.168.202.145@tcp128, 192.168.202.145@tcp129 [14805.293841] LNetError: 462509:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.145 [14805.333051] LNetError: 462509:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [14805.354025] LNetError: Protocol error connecting to 192.168.202.145@tcp106 at host 192.168.202.145:988: is it running a compatible version of Lustre? [14805.378551] LNetError: Skipped 2 previous similar messages [14810.320597] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [14821.725578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14822.002883] Lustre: 465086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14822.110776] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14822.114313] Lustre: Skipped 1 previous similar message [14822.230802] Lustre: lustre-MDT0001: new disk, initializing [14822.289577] Lustre: 465890:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.145@tcp: can't add NID 192.168.202.145@tcp130: rc = -75 [14822.301729] Lustre: 465890:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14822.461128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14822.522970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14822.543291] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14827.350321] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [14829.161221] Lustre: 466226:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.145@tcp: can't add NID 192.168.202.145@tcp130: rc = -75 [14829.171282] Lustre: 466226:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14836.004527] LNetError: 465098:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.145 [14836.033743] LNetError: 465098:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [14836.051111] LNetError: Protocol error connecting to 192.168.202.145@tcp106 at host 192.168.202.145:988: is it running a compatible version of Lustre? [14838.525230] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14839.075452] LNetError: 465091:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.145 [14839.091225] LNetError: Protocol error connecting to 192.168.202.145@tcp117 at host 192.168.202.145:988: is it running a compatible version of Lustre? [14843.170345] LNetError: 465100:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.145 [14843.189497] LNetError: 465100:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [14843.219303] LNetError: Protocol error connecting to 192.168.202.145@tcp128 at host 192.168.202.145:988: is it running a compatible version of Lustre? [14843.231458] LNetError: Skipped 2 previous similar messages [14845.633589] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14851.362996] LNetError: 462508:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.145 [14851.387574] LNetError: 462508:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [14851.406501] LNetError: Protocol error connecting to 192.168.202.145@tcp117 at host 192.168.202.145:988: is it running a compatible version of Lustre? [14851.413610] LNetError: Skipped 1 previous similar message [14853.797758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14854.178832] Lustre: lustre-OST0000: new disk, initializing [14854.183163] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14854.189364] Lustre: 467064:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14854.369940] Lustre: 467069:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.145@tcp: can't add NID 192.168.202.145@tcp130: rc = -75 [14854.378871] Lustre: 467069:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14854.463942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14859.287853] Lustre: 467325:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.145@tcp: can't add NID 192.168.202.145@tcp130: rc = -75 [14859.295922] Lustre: 467325:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14859.408780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14859.425032] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14859.536602] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14860.764479] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [14868.769413] LNetError: 462510:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.145 [14868.790582] LNetError: 462510:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [14868.813749] LNetError: Protocol error connecting to 192.168.202.145@tcp123 at host 192.168.202.145:988: is it running a compatible version of Lustre? [14868.842569] LNetError: Skipped 6 previous similar messages [14870.889795] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14883.302930] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14883.312964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14887.799894] Lustre: server umount lustre-OST0000 complete [14893.541898] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14893.545373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14893.551507] Lustre: Skipped 2 previous similar messages [14893.558128] Lustre: Skipped 2 previous similar messages [14898.827274] Lustre: server umount lustre-MDT0000 complete [14903.076839] LustreError: 465079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790787773 with bad export cookie 1451391675698160882 [14903.081144] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14903.082918] LustreError: 465079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14903.259279] Lustre: server umount lustre-MDT0001 complete [14903.714340] LNetError: 465098:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.145 [14903.719682] LNetError: 465098:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [14903.723602] LNetError: Protocol error connecting to 192.168.202.145@tcp117 at host 192.168.202.145:988: is it running a compatible version of Lustre? [14903.729832] LNetError: Skipped 4 previous similar messages [14916.673463] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [14919.651861] Key type lgssc unregistered [14920.097163] LNet: 468976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14920.111907] LNetError: 468976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14921.120933] LNet: Removed LNI 192.168.202.145@tcp [14921.127966] LNetError: 468976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14921.130906] LNet: Removed LNI 192.168.202.145@tcp100 [14922.208226] LNet: Removed LNI 192.168.202.145@tcp101 [14922.211663] LNetError: 468976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14922.221092] LNetError: 468976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14922.233368] LNet: Removed LNI 192.168.202.145@tcp102 [14923.297282] LNet: Removed LNI 192.168.202.145@tcp103 [14924.320195] LNet: Removed LNI 192.168.202.145@tcp104 [14924.332648] LNetError: 468976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14924.342908] LNetError: 468976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14925.412265] LNet: Removed LNI 192.168.202.145@tcp105 [14926.432156] LNet: Removed LNI 192.168.202.145@tcp106 [14927.456820] LNet: Removed LNI 192.168.202.145@tcp107 [14928.480166] LNet: Removed LNI 192.168.202.145@tcp108 [14928.484145] LNetError: 468976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14928.490564] LNetError: 468976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [14928.498350] LNet: Removed LNI 192.168.202.145@tcp109 [14929.505667] LNet: Removed LNI 192.168.202.145@tcp110 [14930.528736] LNet: Removed LNI 192.168.202.145@tcp111 [14931.552184] LNet: Removed LNI 192.168.202.145@tcp112 [14932.576210] LNet: Removed LNI 192.168.202.145@tcp113 [14933.600166] LNet: Removed LNI 192.168.202.145@tcp114 [14934.688242] LNet: Removed LNI 192.168.202.145@tcp115 [14935.712666] LNet: Removed LNI 192.168.202.145@tcp116 [14936.736173] LNet: Removed LNI 192.168.202.145@tcp117 [14936.738120] LNetError: 468976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14936.765232] LNetError: 468976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14936.777360] LNet: Removed LNI 192.168.202.145@tcp118 [14937.824226] LNet: Removed LNI 192.168.202.145@tcp119 [14938.848275] LNet: Removed LNI 192.168.202.145@tcp120 [14939.936252] LNet: Removed LNI 192.168.202.145@tcp121 [14940.960322] LNet: Removed LNI 192.168.202.145@tcp122 [14941.984186] LNet: Removed LNI 192.168.202.145@tcp123 [14943.008792] LNet: Removed LNI 192.168.202.145@tcp124 [14944.032178] LNet: Removed LNI 192.168.202.145@tcp125 [14945.120219] LNet: Removed LNI 192.168.202.145@tcp126 [14946.144256] LNet: Removed LNI 192.168.202.145@tcp127 [14947.168585] LNet: Removed LNI 192.168.202.145@tcp128 [14948.259327] LNet: Removed LNI 192.168.202.145@tcp129 [14948.263498] LNet: Removed LNI 192.168.202.145@tcp130 [14948.275268] LNet: Removed LNI 192.168.202.145@tcp131 [14948.277181] LNet: Removed LNI 192.168.202.145@tcp132 [14948.290381] LNet: Removed LNI 192.168.202.145@tcp133 [14948.296281] LNet: Removed LNI 192.168.202.145@tcp134 [14948.308534] LNet: Removed LNI 192.168.202.145@tcp135 [14949.384488] Key type .llcrypt unregistered [14949.386579] Key type ._llcrypt unregistered [14976.721211] Key type ._llcrypt registered [14976.722867] Key type .llcrypt registered [14976.887243] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [14995.051467] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [14996.452794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14996.492469] alg: No test for adler32 (adler32-zlib) [14997.563994] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [14997.830539] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [14999.504600] Key type lgssc registered [15001.465425] Lustre: Echo OBD driver; http://www.lustre.org/ [15047.020656] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 13:05:15 (1790787915) [15066.893723] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [15069.390147] Key type lgssc unregistered [15069.802968] LNet: 473081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15069.809898] LNetError: 473081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15069.821290] LNet: Removed LNI 192.168.202.145@tcp [15071.198527] Key type .llcrypt unregistered [15071.201255] Key type ._llcrypt unregistered [15093.603148] Key type ._llcrypt registered [15093.604806] Key type .llcrypt registered [15093.690657] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [15095.247824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15095.281829] alg: No test for adler32 (adler32-zlib) [15096.446889] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [15096.655944] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [15098.360463] Key type lgssc registered [15100.283733] Lustre: Echo OBD driver; http://www.lustre.org/ [15111.853904] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [15111.890539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15113.219885] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15113.297572] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15113.454287] Lustre: lustre-MDT0000: new disk, initializing [15113.541379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15113.608712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15118.830791] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [15132.102766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15132.351577] Lustre: 474311:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15132.469083] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15132.482369] Lustre: Skipped 1 previous similar message [15132.654708] Lustre: lustre-MDT0001: new disk, initializing [15132.913271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15133.014417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15133.044337] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15139.457641] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [15151.019293] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15156.528372] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15165.070901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15165.396670] Lustre: lustre-OST0000: new disk, initializing [15165.398354] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15165.404902] Lustre: 476284:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15165.481593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15170.776908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15170.803336] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15171.035444] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15176.919847] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [15190.584057] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15193.901720] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing lnet_if_list [15198.652937] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15198.675452] LNet: Added LNI 192.168.202.145@tcp10 [8/256/0/180] [15204.292770] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15204.312832] LNet: Added LNI 192.168.202.145@tcp11 [8/256/0/180] [15209.759520] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15209.797842] LNet: Added LNI 192.168.202.145@tcp12 [8/256/0/180] [15215.149635] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15215.189353] LNet: Added LNI 192.168.202.145@tcp13 [8/256/0/180] [15222.029865] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15222.151946] LNet: Added LNI 192.168.202.145@tcp14 [8/256/0/180] [15229.259336] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15229.286768] LNet: Added LNI 192.168.202.145@tcp15 [8/256/0/180] [15238.506788] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15238.570321] LNet: Added LNI 192.168.202.145@tcp16 [8/256/0/180] [15246.319724] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15246.376702] LNet: Added LNI 192.168.202.145@tcp17 [8/256/0/180] [15254.357242] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15254.445110] LNet: Added LNI 192.168.202.145@tcp18 [8/256/0/180] [15261.193054] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15261.282273] LNet: Added LNI 192.168.202.145@tcp19 [8/256/0/180] [15269.623554] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15269.697081] LNet: Added LNI 192.168.202.145@tcp20 [8/256/0/180] [15276.110549] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15276.140888] LNet: Added LNI 192.168.202.145@tcp21 [8/256/0/180] [15282.652773] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15282.684441] LNet: Added LNI 192.168.202.145@tcp22 [8/256/0/180] [15290.055216] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15290.082636] LNet: Added LNI 192.168.202.145@tcp23 [8/256/0/180] [15296.995072] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15297.034102] LNet: Added LNI 192.168.202.145@tcp24 [8/256/0/180] [15303.480330] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15303.512615] LNet: Added LNI 192.168.202.145@tcp25 [8/256/0/180] [15310.753040] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15310.812215] LNet: Added LNI 192.168.202.145@tcp26 [8/256/0/180] [15317.838918] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15317.871662] LNet: Added LNI 192.168.202.145@tcp27 [8/256/0/180] [15324.494554] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15324.519689] LNet: Added LNI 192.168.202.145@tcp28 [8/256/0/180] [15331.725440] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15331.770470] LNet: Added LNI 192.168.202.145@tcp29 [8/256/0/180] [15339.002844] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15339.050944] LNet: Added LNI 192.168.202.145@tcp30 [8/256/0/180] [15346.910274] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15346.953312] LNet: Added LNI 192.168.202.145@tcp31 [8/256/0/180] [15355.051286] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15355.092239] LNet: Added LNI 192.168.202.145@tcp32 [8/256/0/180] [15362.838704] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15362.884175] LNet: Added LNI 192.168.202.145@tcp33 [8/256/0/180] [15371.685330] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15371.724941] LNet: Added LNI 192.168.202.145@tcp34 [8/256/0/180] [15379.597334] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15379.619971] LNet: Added LNI 192.168.202.145@tcp35 [8/256/0/180] [15388.194192] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15388.228653] LNet: Added LNI 192.168.202.145@tcp36 [8/256/0/180] [15396.168486] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15396.209969] LNet: Added LNI 192.168.202.145@tcp37 [8/256/0/180] [15403.865659] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15403.904314] LNet: Added LNI 192.168.202.145@tcp38 [8/256/0/180] [15413.017486] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15413.075460] LNet: Added LNI 192.168.202.145@tcp39 [8/256/0/180] [15422.245789] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15422.269051] LNet: Added LNI 192.168.202.145@tcp40 [8/256/0/180] [15431.910779] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15431.974909] LNet: Added LNI 192.168.202.145@tcp41 [8/256/0/180] [15442.256644] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15442.292567] LNet: Added LNI 192.168.202.145@tcp42 [8/256/0/180] [15450.837685] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15450.860690] LNet: Added LNI 192.168.202.145@tcp43 [8/256/0/180] [15460.816800] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15460.840540] LNet: Added LNI 192.168.202.145@tcp44 [8/256/0/180] [15476.296688] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15476.346706] LNet: Added LNI 192.168.202.145@tcp45 [8/256/0/180] [15488.948650] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15488.982876] LNet: Added LNI 192.168.202.145@tcp46 [8/256/0/180] [15501.621456] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15501.655719] LNet: Added LNI 192.168.202.145@tcp47 [8/256/0/180] [15513.564424] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15513.598269] LNet: Added LNI 192.168.202.145@tcp48 [8/256/0/180] [15524.381427] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15524.441852] LNet: Added LNI 192.168.202.145@tcp49 [8/256/0/180] [15535.603706] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15535.633625] LNet: Added LNI 192.168.202.145@tcp50 [8/256/0/180] [15555.041812] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15555.055556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15555.061296] Lustre: Skipped 2 previous similar messages [15555.075319] Lustre: Skipped 2 previous similar messages [15560.161846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15560.171522] Lustre: Skipped 2 previous similar messages [15561.087861] Lustre: server umount lustre-MDT0000 complete [15566.077615] LustreError: 475109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790788436 with bad export cookie 94306840477159496 [15566.092501] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15566.094060] LustreError: 475109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15567.021925] Lustre: server umount lustre-MDT0001 complete [15586.789433] Lustre: 473573:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790788441/real 1790788441] req@00000000d8cde5af x1877777283486592/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790788457 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15586.826484] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15587.814523] Lustre: 473574:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790788442/real 1790788442] req@0000000067b7e403 x1877777283486720/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790788458 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15589.291813] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [15591.905763] Lustre: 473576:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790788446/real 1790788446] req@00000000495663e6 x1877777283486848/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790788462 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15597.027185] Lustre: 473575:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790788451/real 1790788451] req@00000000f9bd3281 x1877777283487104/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790788467 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15599.509019] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15602.144299] Lustre: 473573:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790788456/real 1790788456] req@0000000022e78c5b x1877777283487232/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790788472 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15610.401034] LustreError: 473572:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000036e6283d x1877777283487872/t0(0) o250->MGC192.168.202.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15610.868575] LustreError: 476275:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15610.887112] LustreError: 476275:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [15610.940361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15611.944176] LustreError: 500539:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15616.351717] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [15617.001582] LustreError: 500538:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15617.015906] LustreError: 500538:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [15621.106710] LustreError: 500537:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15621.127846] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15621.143591] LustreError: 500537:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15626.169587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15626.212558] LustreError: 500538:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15626.664636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15631.875527] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15633.032393] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [15643.127310] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15651.122809] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15677.409404] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15677.414527] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15677.436271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15677.930529] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15680.670944] Lustre: server umount lustre-OST0000 complete [15688.163354] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15688.164545] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15688.184109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15688.184120] Lustre: Skipped 1 previous similar message [15688.237479] Lustre: Skipped 2 previous similar messages [15691.634417] Lustre: server umount lustre-MDT0000 complete [15693.284400] LustreError: 500542:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15693.319380] LustreError: 500542:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [15696.339583] LustreError: 474306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790788567 with bad export cookie 94306840477162177 [15696.348295] LustreError: 474306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15696.356467] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15696.813661] Lustre: server umount lustre-MDT0001 complete [15711.593651] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [15715.292248] Key type lgssc unregistered [15715.745698] LNet: 503342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15715.753592] LNetError: 503342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15716.768265] LNet: Removed LNI 192.168.202.145@tcp [15716.774252] LNetError: 503342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15716.784289] LNet: Removed LNI 192.168.202.145@tcp10 [15716.793558] LNet: Removed LNI 192.168.202.145@tcp11 [15716.798662] LNet: Removed LNI 192.168.202.145@tcp12 [15716.806078] LNet: Removed LNI 192.168.202.145@tcp13 [15716.808053] LNet: Removed LNI 192.168.202.145@tcp14 [15716.814690] LNet: Removed LNI 192.168.202.145@tcp15 [15716.819356] LNet: Removed LNI 192.168.202.145@tcp16 [15716.844880] LNet: Removed LNI 192.168.202.145@tcp17 [15716.846865] LNet: Removed LNI 192.168.202.145@tcp18 [15716.862174] LNet: Removed LNI 192.168.202.145@tcp19 [15716.864337] LNet: Removed LNI 192.168.202.145@tcp20 [15716.869514] LNet: Removed LNI 192.168.202.145@tcp21 [15716.874313] LNet: Removed LNI 192.168.202.145@tcp22 [15716.879316] LNet: Removed LNI 192.168.202.145@tcp23 [15716.885966] LNet: Removed LNI 192.168.202.145@tcp24 [15716.889572] LNet: Removed LNI 192.168.202.145@tcp25 [15716.898798] LNet: Removed LNI 192.168.202.145@tcp26 [15716.903521] LNet: Removed LNI 192.168.202.145@tcp27 [15716.908648] LNet: Removed LNI 192.168.202.145@tcp28 [15716.917678] LNet: Removed LNI 192.168.202.145@tcp29 [15716.919724] LNet: Removed LNI 192.168.202.145@tcp30 [15716.923202] LNet: Removed LNI 192.168.202.145@tcp31 [15716.927819] LNet: Removed LNI 192.168.202.145@tcp32 [15716.931960] LNet: Removed LNI 192.168.202.145@tcp33 [15716.935592] LNet: Removed LNI 192.168.202.145@tcp34 [15716.940176] LNet: Removed LNI 192.168.202.145@tcp35 [15716.946239] LNet: Removed LNI 192.168.202.145@tcp36 [15716.949846] LNet: Removed LNI 192.168.202.145@tcp37 [15716.954273] LNet: Removed LNI 192.168.202.145@tcp38 [15716.964788] LNet: Removed LNI 192.168.202.145@tcp39 [15716.968333] LNet: Removed LNI 192.168.202.145@tcp40 [15716.979773] LNet: Removed LNI 192.168.202.145@tcp41 [15716.988191] LNet: Removed LNI 192.168.202.145@tcp42 [15716.993250] LNet: Removed LNI 192.168.202.145@tcp43 [15716.995659] LNet: Removed LNI 192.168.202.145@tcp44 [15717.010818] LNet: Removed LNI 192.168.202.145@tcp45 [15717.026967] LNet: Removed LNI 192.168.202.145@tcp46 [15717.031286] LNet: Removed LNI 192.168.202.145@tcp47 [15717.035269] LNet: Removed LNI 192.168.202.145@tcp48 [15717.042929] LNet: Removed LNI 192.168.202.145@tcp49 [15717.054808] LNet: Removed LNI 192.168.202.145@tcp50 [15718.000545] Key type .llcrypt unregistered [15718.003067] Key type ._llcrypt unregistered [15742.706918] Key type ._llcrypt registered [15742.708958] Key type .llcrypt registered [15742.888990] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [15764.135471] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [15765.733590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15765.806141] alg: No test for adler32 (adler32-zlib) [15767.151313] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [15767.637762] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [15769.488703] Key type lgssc registered [15771.320349] Lustre: Echo OBD driver; http://www.lustre.org/ [15820.109410] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [15820.132548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15821.484411] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15821.558111] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15821.699674] Lustre: lustre-MDT0000: new disk, initializing [15821.816986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15821.850197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15827.358194] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [15840.819190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15841.007711] Lustre: 506558:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15841.116420] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15841.125418] Lustre: Skipped 1 previous similar message [15841.219173] Lustre: lustre-MDT0001: new disk, initializing [15841.343691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15841.399976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15841.429882] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15848.415782] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [15860.559798] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15866.885417] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15876.905839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15877.233941] Lustre: *** cfs_fail_loc=90f, val=0*** [15877.409213] Lustre: lustre-OST0000: new disk, initializing [15877.417095] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15877.428654] Lustre: 508576:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15877.532641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15883.333392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15883.341484] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15883.488433] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15885.417368] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [15896.938265] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15909.351826] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15909.359186] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15909.364804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15909.378933] Lustre: Skipped 1 previous similar message [15909.407521] Lustre: Skipped 1 previous similar message [15911.773896] Lustre: server umount lustre-OST0000 complete [15918.056582] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15918.069493] LustreError: Skipped 1 previous similar message [15918.082348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15918.111610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15919.587611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15919.593673] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15923.198705] Lustre: server umount lustre-MDT0000 complete [15927.631729] LustreError: 507352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790788798 with bad export cookie 13613196076483589153 [15927.635673] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15927.642482] LustreError: 507352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15928.178958] Lustre: server umount lustre-MDT0001 complete [15940.631062] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [15944.542493] Key type lgssc unregistered [15945.017811] LNet: 510482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15945.025781] LNetError: 510482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15945.054791] LNet: Removed LNI 192.168.202.145@tcp [15946.260090] Key type .llcrypt unregistered [15946.262188] Key type ._llcrypt unregistered [15969.634298] Key type ._llcrypt registered [15969.638739] Key type .llcrypt registered [15969.768613] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [15971.288878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15971.313609] alg: No test for adler32 (adler32-zlib) [15972.673574] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [15973.022866] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [15974.760287] Key type lgssc registered [15976.218741] Lustre: Echo OBD driver; http://www.lustre.org/ [15983.728530] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [15983.764714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15985.145624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15990.203254] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [15994.380649] LustreError: 511785:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15994.410223] LustreError: 511785:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [15999.458899] LustreError: 511784:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16002.314086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16002.942199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16009.045788] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [16018.497667] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16026.307136] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16035.562976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16036.109245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16044.057800] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [16053.549872] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16066.258837] loop0: detected capacity change from 0 to 204800000 [16066.439668] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16066.471185] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16066.502993] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16075.001076] loop0: detected capacity change from 0 to 204800000 [16075.171096] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16075.196702] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16075.247477] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16082.121386] loop0: detected capacity change from 0 to 204800000 [16088.732640] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16089.067047] Lustre: fs15246-OST0000: new disk, initializing [16089.072426] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16089.093294] Lustre: 515379:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16089.263659] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16098.566986] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [16109.984666] loop1: detected capacity change from 0 to 204800000 [16117.369220] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16117.554644] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16117.660579] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16117.752929] Lustre: fs15246-MDT0000: new disk, initializing [16117.915303] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16117.957448] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16118.069445] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16119.137548] Lustre: 516502:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000009170aeb2 x1877778201891328/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1790789005 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16123.023909] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [16128.238683] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16128.260563] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16128.294151] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16149.997209] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16150.023519] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16153.964032] Lustre: server umount fs15246-MDT0000 complete [16155.108954] LustreError: 513189:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16160.236669] LustreError: 511781:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16164.326686] Lustre: server umount fs15246-OST0000 complete [16180.708954] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16180.720842] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16180.735213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16180.744802] Lustre: Skipped 1 previous similar message [16184.688968] Lustre: server umount lustre-OST0000 complete [16190.951164] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16190.972327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16190.991171] Lustre: Skipped 2 previous similar messages [16191.005726] Lustre: Skipped 1 previous similar message [16196.066334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16196.079306] Lustre: Skipped 1 previous similar message [16201.188416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16201.197545] Lustre: Skipped 1 previous similar message [16203.232188] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16203.420876] Lustre: server umount lustre-MDT0000 complete [16206.313601] LustreError: 511784:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16206.342209] LustreError: 511784:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [16208.352243] LustreError: 511763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790789079 with bad export cookie 17350837833275734724 [16208.363817] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16208.371260] LustreError: 511763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16209.001883] Lustre: server umount lustre-MDT0001 complete [16222.578246] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [16226.216500] Key type lgssc unregistered [16226.648186] LNet: 519601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16226.657430] LNetError: 519601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16226.690653] LNet: Removed LNI 192.168.202.145@tcp [16228.025618] Key type .llcrypt unregistered [16228.028660] Key type ._llcrypt unregistered [16250.748472] Key type ._llcrypt registered [16250.751251] Key type .llcrypt registered [16250.906611] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [16252.571501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16252.615974] alg: No test for adler32 (adler32-zlib) [16253.773146] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [16254.023930] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [16255.736405] Key type lgssc registered [16257.012111] Lustre: Echo OBD driver; http://www.lustre.org/ [16306.516868] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [16319.520082] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [16365.652901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [16365.693482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16367.040838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16367.090804] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16367.155247] Lustre: lustre-MDT0000: new disk, initializing [16367.215057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16367.229863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16372.912352] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [16384.647847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16384.805476] Lustre: 524087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16384.878493] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16384.885081] Lustre: Skipped 1 previous similar message [16384.970684] Lustre: lustre-MDT0001: new disk, initializing [16385.097523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16385.150487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16385.164286] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16390.118139] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [16401.166923] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16409.000714] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16417.203514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16417.692276] Lustre: lustre-OST0000: new disk, initializing [16417.698232] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16417.704394] Lustre: 526055:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16417.816241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16425.107945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16425.118994] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16425.287191] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16425.820641] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [16436.486817] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16441.640802] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16442.059416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16446.892410] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16447.180550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16451.041762] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16451.042575] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16451.047029] LustreError: Skipped 1 previous similar message [16451.058207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16451.065596] Lustre: Skipped 1 previous similar message [16451.087299] Lustre: Skipped 1 previous similar message [16456.179158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16456.192285] Lustre: Skipped 1 previous similar message [16461.282775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16461.302445] Lustre: Skipped 1 previous similar message [16463.328251] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16464.044087] Lustre: server umount lustre-OST0000 complete [16471.521830] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16471.535496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16471.549456] Lustre: Skipped 1 previous similar message [16476.047500] Lustre: server umount lustre-MDT0000 complete [16481.723949] LustreError: 524079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790789352 with bad export cookie 4370835568385719922 [16481.736616] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16481.751034] LustreError: 524079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16482.652941] Lustre: server umount lustre-MDT0001 complete [16511.004724] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [16524.268720] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [16563.851585] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16283 sec ========= 13:30:31 (1790789431) [16565.908742] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:30:34 (1790789434) === [16570.170669] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:30:38 (1790789438) === [16618.772462] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [16621.792890] Key type lgssc unregistered [16622.030556] LNet: 532705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16622.036875] LNetError: 532705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16622.059272] LNet: Removed LNI 192.168.202.145@tcp [16622.824758] Key type .llcrypt unregistered [16622.826895] Key type ._llcrypt unregistered