[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 512021059 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2891144K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001018] APIC: Switch to symmetric I/O mode setup [ 0.002409] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004008] kvm-guest: setup PV IPIs [ 0.006581] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008014] pid_max: default: 32768 minimum: 301 [ 0.009124] LSM: Security Framework initializing [ 0.010053] Yama: becoming mindful. [ 0.011032] SELinux: Initializing. [ 0.012058] *** VALIDATE selinux *** [ 0.019379] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023399] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024131] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025137] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026123] *** VALIDATE tmpfs *** [ 0.027248] *** VALIDATE proc *** [ 0.028255] *** VALIDATE cgroup *** [ 0.029008] *** VALIDATE cgroup2 *** [ 0.029931] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.030000] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.030000] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.030032] Spectre V2 : User space: Vulnerable [ 0.030864] Speculative Store Bypass: Vulnerable [ 0.032694] debug: unmapping init [mem 0xffffffffb3659000-0xffffffffb3660fff] [ 0.034157] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.035637] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.036020] ... version: 2 [ 0.037011] ... bit width: 48 [ 0.038011] ... generic registers: 4 [ 0.039019] ... value mask: 0000ffffffffffff [ 0.040026] ... max period: 00007fffffffffff [ 0.041012] ... fixed-purpose events: 3 [ 0.041770] ... event mask: 000000070000000f [ 0.042268] rcu: Hierarchical SRCU implementation. [ 0.044447] smp: Bringing up secondary CPUs ... [ 0.045501] x86: Booting SMP configuration: [ 0.046023] .... node #0, CPUs: #1 #2 #3 [ 0.049497] smp: Brought up 1 node, 4 CPUs [ 0.050990] smpboot: Max logical packages: 1 [ 0.051022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.134026] node 0 deferred pages initialised in 80ms [ 0.139157] devtmpfs: initialized [ 0.140360] x86/mm: Memory block size: 128MB [ 0.144598] gcov: version magic: 0x41383552 [ 0.146384] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.150102] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.152403] pinctrl core: initialized pinctrl subsystem [ 0.155213] [ 0.155737] ************************************************************* [ 0.158018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161020] ** ** [ 0.163015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.166014] ** ** [ 0.168014] ** This means that this kernel is built to expose internal ** [ 0.169016] ** IOMMU data structures, which may compromise security on ** [ 0.173019] ** your system. ** [ 0.175013] ** ** [ 0.178015] ** If you see this message and you are not debugging the ** [ 0.180017] ** kernel, report this immediately to your vendor! ** [ 0.183016] ** ** [ 0.186018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.188014] ************************************************************* [ 0.191937] NET: Registered protocol family 16 [ 0.194441] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.197088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.199058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.204037] cpuidle: using governor menu [ 0.206116] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.207273] PCI: Using configuration type 1 for base access [ 0.210184] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.219127] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.220774] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.224409] cryptd: max_cpu_qlen set to 1000 [ 0.227290] ACPI: Added _OSI(Module Device) [ 0.228023] ACPI: Added _OSI(Processor Device) [ 0.230031] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.231013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.236251] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.241626] ACPI: Interpreter enabled [ 0.242000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.244016] ACPI: Using IOAPIC for interrupt routing [ 0.245092] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.247386] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.256316] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.257043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.259013] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.260101] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.264055] acpiphp: Slot [2] registered [ 0.265043] acpiphp: Slot [5] registered [ 0.266071] acpiphp: Slot [6] registered [ 0.267060] acpiphp: Slot [7] registered [ 0.267993] acpiphp: Slot [8] registered [ 0.269073] acpiphp: Slot [9] registered [ 0.270086] acpiphp: Slot [10] registered [ 0.271026] acpiphp: Slot [3] registered [ 0.271957] acpiphp: Slot [4] registered [ 0.273125] acpiphp: Slot [11] registered [ 0.274081] acpiphp: Slot [12] registered [ 0.274973] acpiphp: Slot [13] registered [ 0.276114] acpiphp: Slot [14] registered [ 0.277071] acpiphp: Slot [15] registered [ 0.277989] acpiphp: Slot [16] registered [ 0.279075] acpiphp: Slot [17] registered [ 0.279898] acpiphp: Slot [18] registered [ 0.281061] acpiphp: Slot [19] registered [ 0.281970] acpiphp: Slot [20] registered [ 0.282119] acpiphp: Slot [21] registered [ 0.283027] acpiphp: Slot [22] registered [ 0.283894] acpiphp: Slot [23] registered [ 0.285087] acpiphp: Slot [24] registered [ 0.286142] acpiphp: Slot [25] registered [ 0.287083] acpiphp: Slot [26] registered [ 0.288000] acpiphp: Slot [27] registered [ 0.289141] acpiphp: Slot [28] registered [ 0.291109] acpiphp: Slot [29] registered [ 0.292166] acpiphp: Slot [30] registered [ 0.293158] acpiphp: Slot [31] registered [ 0.294073] PCI host bridge to bus 0000:00 [ 0.296021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.298024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.300023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.303034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.306029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.309030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.311197] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.314088] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.318484] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.328016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.334223] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.337018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.340020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.342019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.345429] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.347695] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.350046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.352598] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.357022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.365889] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.369000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.373168] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.378013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.382011] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.397020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.407276] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.417019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.424018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.444020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.454998] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.464016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.472019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.489019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.499855] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.506020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.515015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.533020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.542842] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.550019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.560019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.577019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.590882] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.603034] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.608029] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.624088] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.633584] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.636296] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.637289] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.639263] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.641205] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.645067] iommu: Default domain type: Passthrough [ 0.648638] SCSI subsystem initialized [ 0.650194] ACPI: bus type USB registered [ 0.652119] usbcore: registered new interface driver usbfs [ 0.653121] usbcore: registered new interface driver hub [ 0.655113] usbcore: registered new device driver usb [ 0.657315] pps_core: LinuxPPS API ver. 1 registered [ 0.659013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.662064] PTP clock support registered [ 0.665089] EDAC MC: Ver: 3.0.0 [ 0.667133] PCI: Using ACPI for IRQ routing [ 0.669262] NetLabel: Initializing [ 0.670000] NetLabel: domain hash size = 128 [ 0.673014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.676135] NetLabel: unlabeled traffic allowed by default [ 0.678134] vgaarb: loaded [ 0.679318] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.681018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.687787] clocksource: Switched to clocksource kvm-clock [ 0.801064] VFS: Disk quotas dquot_6.6.0 [ 0.802448] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.804166] *** VALIDATE ramfs *** [ 0.805443] *** VALIDATE hugetlbfs *** [ 0.806494] pnp: PnP ACPI init [ 0.808542] pnp: PnP ACPI: found 6 devices [ 0.845119] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.848031] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.849333] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.851275] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.853129] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.854886] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.857243] NET: Registered protocol family 2 [ 0.858858] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.862081] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.864970] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.869733] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.873480] TCP: Hash tables configured (established 65536 bind 65536) [ 0.876643] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.879748] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.882733] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.885710] NET: Registered protocol family 1 [ 0.888634] RPC: Registered named UNIX socket transport module. [ 0.890831] RPC: Registered udp transport module. [ 0.893020] RPC: Registered tcp transport module. [ 0.895402] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.898784] NET: Registered protocol family 44 [ 0.900632] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.902972] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.905361] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.908077] PCI: CLS 0 bytes, default 64 [ 0.910725] Unpacking initramfs... [ 2.312924] debug: unmapping init [mem 0xffff8b11bcc54000-0xffff8b11bffbffff] [ 2.316962] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.319248] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.322163] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.823475] Initialise system trusted keyrings [ 2.824908] Key type blacklist registered [ 2.826780] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.835905] zbud: loaded [ 2.839067] *** VALIDATE nfs *** [ 2.840321] *** VALIDATE nfs4 *** [ 2.841828] pstore: using deflate compression [ 2.845852] Platform Keyring initialized [ 2.924823] NET: Registered protocol family 38 [ 2.925951] Key type asymmetric registered [ 2.927069] Asymmetric key parser 'x509' registered [ 2.928600] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.930724] io scheduler mq-deadline registered [ 2.931707] io scheduler kyber registered [ 2.932765] io scheduler bfq registered [ 2.933818] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.935621] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.937179] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.939270] ACPI: Power Button [PWRF] [ 2.942548] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.947080] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.028155] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.041160] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.067226] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.094616] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.122149] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.126670] Non-volatile memory driver v1.3 [ 3.128593] Linux agpgart interface v0.103 [ 3.161638] virtio_blk virtio1: [vda] 149952 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.164634] vda: detected capacity change from 0 to 76775424 [ 3.186342] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.189073] vdb: detected capacity change from 0 to 1073741824 [ 3.210084] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.212940] vdc: detected capacity change from 0 to 2621440000 [ 3.228507] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.231428] vdd: detected capacity change from 0 to 2621440000 [ 3.253740] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.255940] vde: detected capacity change from 0 to 4294967296 [ 3.276807] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.280089] vdf: detected capacity change from 0 to 4294967296 [ 3.293584] libphy: Fixed MDIO Bus: probed [ 3.302367] usbcore: registered new interface driver usbserial_generic [ 3.304764] usbserial: USB Serial support registered for generic [ 3.307086] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.311505] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.313354] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.316275] mousedev: PS/2 mouse device common for all mice [ 3.318671] rtc_cmos 00:05: RTC can wake from S4 [ 3.321278] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.324365] rtc_cmos 00:05: registered as rtc0 [ 3.326660] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.330587] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.331075] intel_pstate: CPU model not supported [ 3.336616] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.339914] hid: raw HID events driver (C) Jiri Kosina [ 3.342800] usbcore: registered new interface driver usbhid [ 3.344719] usbhid: USB HID core driver [ 3.346177] drop_monitor: Initializing network drop monitor service [ 3.348487] Initializing XFRM netlink socket [ 3.350399] NET: Registered protocol family 10 [ 3.354151] Segment Routing with IPv6 [ 3.355922] NET: Registered protocol family 17 [ 3.357977] mpls_gso: MPLS GSO support [ 3.362600] RAS: Correctable Errors collector initialized. [ 3.363902] AVX version of gcm_enc/dec engaged. [ 3.365224] AES CTR mode by8 optimization enabled [ 3.439268] sched_clock: Marking stable (3439173893, 0)->(4354074104, -914900211) [ 3.442944] registered taskstats version 1 [ 3.445303] Loading compiled-in X.509 certificates [ 3.447648] zswap: loaded using pool lzo/zbud [ 3.475725] Key type big_key registered [ 3.490422] Key type encrypted registered [ 3.492597] ima: No TPM chip found, activating TPM-bypass! [ 3.494894] ima: Allocated hash algorithm: sha1 [ 3.496875] ima: No architecture policies found [ 3.498843] evm: Initialising EVM extended attributes: [ 3.500205] evm: security.selinux [ 3.501423] evm: security.ima [ 3.502607] evm: security.capability [ 3.503990] evm: HMAC attrs: 0x1 [ 3.506597] rtc_cmos 00:05: setting system clock to 2026-09-10 17:59:00 UTC (1789063140) [ 3.513444] debug: unmapping init [mem 0xffffffffb4603000-0xffffffffb47fffff] [ 3.516807] debug: unmapping init [mem 0xffffffffb3382000-0xffffffffb3658fff] [ 3.526098] Write protecting the kernel read-only data: 28672k [ 3.529553] debug: unmapping init [mem 0xffffffffb1a03000-0xffffffffb1bfffff] [ 3.532298] debug: unmapping init [mem 0xffffffffb2314000-0xffffffffb23fffff] [ 3.563113] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.568544] systemd[1]: Detected virtualization kvm. [ 3.569570] systemd[1]: Detected architecture x86-64. [ 3.570598] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.593422] systemd[1]: No hostname configured. [ 3.594839] systemd[1]: Set hostname to . [ 3.596266] random: systemd: uninitialized urandom read (16 bytes read) [ 3.598698] systemd[1]: Initializing machine ID from random generator. [ 3.657129] random: ln: uninitialized urandom read (6 bytes read) [ 3.731440] random: systemd: uninitialized urandom read (16 bytes read) [ 3.733152] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.736654] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.739767] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Reached target Sockets. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.297813] device-mapper: uevent: version 1.0.3 [ 4.299545] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.991355] virtio_net virtio0 ens2: renamed from eth0 [ 5.017110] random: fast init done [ 5.092448] scsi host0: ata_piix [ 5.132209] scsi host1: ata_piix [ 5.133667] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.136255] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.509422] dracut-initqueue[593]: RTNETLINK answers: File exists [ 10.010693] random: crng init done [ 10.012103] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.472782] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.755286] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.969731] SELinux: Disabled at runtime. [ 12.042684] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.047847] systemd[1]: Detected virtualization kvm. [ 12.050189] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.635974] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.641315] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.648748] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.656517] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.660607] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.668858] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.676338] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ 12.781403] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.249115] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.698622] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.733727] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.922349] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.937786] EDAC sbridge: Ver: 1.1.2 [ 15.842205] Key type dns_resolver registered [ 16.245589] NFS: Registering the id_resolver key type [ 16.246988] Key type id_resolver registered [ 16.248141] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg202-server login: [ 57.981549] libcfs: loading out-of-tree module taints kernel. [ 58.034618] Key type ._llcrypt registered [ 58.038727] Key type .llcrypt registered [ 58.150640] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 89.699950] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 92.355154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 92.397222] alg: No test for adler32 (adler32-zlib) [ 94.499839] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 95.547979] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 97.279374] Key type lgssc registered [ 99.274763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 115.726658] hrtimer: interrupt took 17559347 ns [ 122.448301] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 172.151082] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 188.921621] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 189.004256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 190.452742] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 190.498595] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 190.637484] Lustre: lustre-MDT0000: new disk, initializing [ 190.799364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 190.847387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 196.782982] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.669259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 212.770685] Lustre: 6527:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 212.797303] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 212.800237] Lustre: Skipped 1 previous similar message [ 212.890591] Lustre: lustre-MDT0001: new disk, initializing [ 213.007381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 213.037133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 213.048115] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 218.439538] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.995185] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 235.317416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 235.682183] Lustre: lustre-OST0000: new disk, initializing [ 235.686743] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 235.691916] Lustre: 8465:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 235.764630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 235.934093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 235.957289] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 236.004375] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 242.888506] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 258.825581] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 258.955579] Lustre: lustre-OST0001: new disk, initializing [ 258.959209] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 258.963750] Lustre: 9539:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 259.030807] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 266.564414] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 266.821877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 266.829306] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 266.988738] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 280.548714] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 289.309479] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 296.756996] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing check_logdir /tmp/testlogs/ [ 302.231280] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing yml_node [ 307.196512] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 309.799693] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 312.885455] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 314.872558] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 10 14:04:10 EDT 2026 [ 334.208224] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 336.435853] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 356.341033] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 356.361913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 356.392209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 359.393337] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 359.395930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 359.415917] Lustre: Skipped 1 previous similar message [ 359.434324] Lustre: Skipped 3 previous similar messages [ 360.937106] Lustre: server umount lustre-MDT0000 complete [ 365.311563] LustreError: 6521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789063502 with bad export cookie 13537294147729998759 [ 365.328161] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 369.632615] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 369.633946] LustreError: 6536:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 369.634159] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 369.656232] Lustre: Skipped 2 previous similar messages [ 369.665263] LustreError: 6536:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 370.656249] LustreError: 6535:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 370.674943] LustreError: 6535:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 371.805313] Lustre: server umount lustre-MDT0001 complete [ 375.949290] Lustre: server umount lustre-OST0000 complete [ 380.139248] Lustre: server umount lustre-OST0001 complete [ 387.144612] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 398.263116] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 441.682964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 442.121045] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 442.167473] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 442.281658] Lustre: lustre-MDT0000: new disk, initializing [ 442.453484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 442.478280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 447.225201] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 459.475898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 459.751680] Lustre: 15096:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 459.836585] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 459.840261] Lustre: Skipped 1 previous similar message [ 459.978343] Lustre: lustre-MDT0001: new disk, initializing [ 460.172414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 460.251544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 460.267312] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 467.132744] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 478.348492] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 485.004515] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 492.788162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 493.003719] Lustre: lustre-OST0000: new disk, initializing [ 493.008909] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 493.013965] Lustre: 17065:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 493.087546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 494.255274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 494.286322] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 494.383159] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 500.093516] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 512.647246] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 519.461887] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 519.765389] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 526.106568] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 526.352178] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 530.401165] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 530.408177] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 530.416845] LustreError: Skipped 1 previous similar message [ 530.431441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 530.446572] Lustre: Skipped 1 previous similar message [ 530.482246] Lustre: Skipped 4 previous similar messages [ 534.967828] Lustre: server umount lustre-OST0000 complete [ 540.642463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 540.672687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 540.702521] Lustre: Skipped 1 previous similar message [ 545.779443] LustreError: 15938:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 545.813363] LustreError: 15938:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 547.249267] Lustre: server umount lustre-MDT0000 complete [ 550.891428] LustreError: 15108:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.919235] LustreError: 15108:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 551.984901] LustreError: 15088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789063688 with bad export cookie 13537294147730000292 [ 551.994432] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 551.995022] LustreError: 15088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 552.520663] Lustre: server umount lustre-MDT0001 complete [ 558.002366] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 14:08:13 (1789063693) [ 561.420675] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 564.857409] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 14:08:19 (1789063699) [ 571.552261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 572.329474] LustreError: 19477:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 572.487913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 577.162537] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 582.638242] LustreError: 19478:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 582.667133] LustreError: 19478:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 586.158964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 591.670862] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 599.291939] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 605.921906] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 613.624351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 620.976093] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 629.096285] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 642.017084] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 642.018738] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 642.038475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 642.054931] Lustre: Skipped 2 previous similar messages [ 642.065417] Lustre: Skipped 1 previous similar message [ 645.943591] Lustre: server umount lustre-OST0000 complete [ 662.499330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 662.505913] Lustre: Skipped 5 previous similar messages [ 664.032069] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 664.624995] Lustre: server umount lustre-MDT0000 complete [ 667.616984] LustreError: 19473:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 667.649461] LustreError: 19473:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 669.616553] LustreError: 19458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789063806 with bad export cookie 13537294147730001167 [ 669.635566] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 670.151256] Lustre: server umount lustre-MDT0001 complete [ 684.772758] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 689.156408] Key type lgssc unregistered [ 689.558560] LNet: 23014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 689.573525] LNetError: 23014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 689.592505] LNet: Removed LNI 192.168.202.102@tcp [ 690.738169] Key type .llcrypt unregistered [ 690.739651] Key type ._llcrypt unregistered [ 719.941990] Key type ._llcrypt registered [ 719.944288] Key type .llcrypt registered [ 720.014200] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 721.931520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 722.007075] alg: No test for adler32 (adler32-zlib) [ 723.406834] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 723.794352] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 725.519594] Key type lgssc registered [ 727.134856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 737.946764] loop: module loaded [ 738.100138] loop0: detected capacity change from 0 to 204800000 [ 738.401024] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 738.417477] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 738.734438] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 746.167741] loop0: detected capacity change from 0 to 204800000 [ 746.510826] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 746.530361] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 746.547498] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 753.094258] loop0: detected capacity change from 0 to 204800000 [ 760.602566] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 760.646184] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 762.080232] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 762.169653] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 762.290755] Lustre: lustre-MDT0000: new disk, initializing [ 762.399678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 762.451095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 767.780116] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 778.013482] loop1: detected capacity change from 0 to 204800000 [ 785.532185] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 785.982588] Lustre: lustre-OST07c6: new disk, initializing [ 785.987303] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 785.996031] Lustre: Skipped 1 previous similar message [ 786.002122] Lustre: 26217:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 786.102461] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 793.480431] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 793.713655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 793.723443] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 793.763615] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 800.492038] Lustre: 25205:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 800.502582] Lustre: 25205:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 800.517490] Lustre: 25205:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 800.521853] Lustre: 25205:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 800.527774] Lustre: 25205:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 800.532600] Lustre: 25205:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 824.295663] Lustre: 25204:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 824.304952] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 824.312656] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 824.330695] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 824.342168] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 824.353369] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 829.428729] Lustre: 25204:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 829.444453] Lustre: 25204:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 829.456563] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 829.467665] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 829.484375] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 829.496489] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 829.518976] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 829.527512] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 829.535752] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 829.547021] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 829.557956] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 829.566074] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 832.816393] Lustre: 25205:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 832.832061] Lustre: 25205:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 832.842435] Lustre: 25205:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 832.860304] Lustre: 25205:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 832.871643] Lustre: 25205:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 832.880533] Lustre: 25205:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 832.886559] Lustre: 25205:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 832.891553] Lustre: 25205:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 832.898686] Lustre: 25205:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 832.906541] Lustre: 25205:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 832.911667] Lustre: 25205:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 832.925257] Lustre: 25205:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 838.722080] Lustre: 26733:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 838.752691] Lustre: 26733:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 838.774624] Lustre: 26733:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 838.790380] Lustre: 26733:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 838.813756] Lustre: 26733:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 838.825425] Lustre: 26733:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 838.840904] Lustre: 26733:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 838.867541] Lustre: 26733:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 838.886391] Lustre: 26733:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 838.899793] Lustre: 26733:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 838.919352] Lustre: 26733:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 838.934670] Lustre: 26733:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 847.320168] Lustre: 25205:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 847.347371] Lustre: 25205:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 847.368185] Lustre: 25205:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 847.386368] Lustre: 25205:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 847.390112] Lustre: 25205:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 847.395194] Lustre: 25205:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 847.403269] Lustre: 25205:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 847.410128] Lustre: 25205:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 847.416161] Lustre: 25205:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 847.423351] Lustre: 25205:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 847.434583] Lustre: 25205:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 847.445309] Lustre: 25205:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 901.092612] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 901.134295] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 901.135040] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 904.537150] Lustre: server umount lustre-OST07c6 complete [ 916.705080] Lustre: server umount lustre-MDT0000 complete [ 934.569515] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 937.189915] Key type lgssc unregistered [ 937.492609] LNet: 28785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 937.498414] LNetError: 28785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 937.509907] LNet: Removed LNI 192.168.202.102@tcp [ 938.338434] Key type .llcrypt unregistered [ 938.340811] Key type ._llcrypt unregistered [ 960.241066] Key type ._llcrypt registered [ 960.243973] Key type .llcrypt registered [ 960.371450] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 961.836938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 961.968090] alg: No test for adler32 (adler32-zlib) [ 963.273600] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 963.689910] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 965.391245] Key type lgssc registered [ 966.553839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 975.786668] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 975.827211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 977.377830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 982.747228] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 987.622180] LustreError: 30052:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 987.652691] LustreError: 30052:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 992.738506] LustreError: 30053:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 992.806878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 993.475454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 999.653280] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1008.685837] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1015.727783] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1023.430764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1024.094696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1031.708318] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1041.253114] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1047.691690] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1070.563109] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1070.576140] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1070.592861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1073.637558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1073.641278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1073.653707] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1078.752657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1078.766815] Lustre: Skipped 2 previous similar messages [ 1083.359368] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1083.561305] Lustre: server umount lustre-OST0000 complete [ 1088.992765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1089.013068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1089.025660] Lustre: Skipped 1 previous similar message [ 1089.047453] Lustre: Skipped 1 previous similar message [ 1094.989685] Lustre: server umount lustre-MDT0000 complete [ 1097.185224] LustreError: 30053:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1097.197441] LustreError: 30053:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1099.689832] LustreError: 30032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789064236 with bad export cookie 11719726344775046689 [ 1099.704667] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1099.720833] LustreError: 30032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1100.329764] Lustre: server umount lustre-MDT0001 complete [ 1114.129184] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 1117.649668] Key type lgssc unregistered [ 1118.079888] LNet: 33812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1118.084462] LNetError: 33812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1118.127347] LNet: Removed LNI 192.168.202.102@tcp [ 1119.202610] Key type .llcrypt unregistered [ 1119.205170] Key type ._llcrypt unregistered [ 1140.619139] Key type ._llcrypt registered [ 1140.621525] Key type .llcrypt registered [ 1140.701414] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 1142.200476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1142.428137] alg: No test for adler32 (adler32-zlib) [ 1143.656136] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 1143.946925] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 1145.647489] Key type lgssc registered [ 1147.218777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1155.884248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1155.912878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1157.806253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1163.506733] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1167.333164] LustreError: 35081:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1167.353359] LustreError: 35081:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1172.455623] LustreError: 35080:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1173.275388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1173.706605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1179.382267] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1188.786838] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1196.070876] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1204.235744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1204.723088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1212.866776] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1221.928083] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1227.268465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1242.597465] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1242.598427] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1242.621594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1242.658196] Lustre: Skipped 1 previous similar message [ 1245.750234] Lustre: server umount lustre-OST0000 complete [ 1250.275610] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1250.283187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1250.295726] Lustre: Skipped 1 previous similar message [ 1250.303313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1252.845986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1252.862131] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1256.751452] Lustre: server umount lustre-MDT0000 complete [ 1257.964699] LustreError: 35809:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1257.980383] LustreError: 35809:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1261.914756] LustreError: 35061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789064398 with bad export cookie 1920978367483357575 [ 1261.937936] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1261.940037] LustreError: 35061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1262.353632] Lustre: server umount lustre-MDT0001 complete [ 1276.015141] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 1279.824847] Key type lgssc unregistered [ 1280.248397] LNet: 38639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1280.262555] LNetError: 38639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1281.317640] LNet: Removed LNI 192.168.202.102@tcp [ 1282.408384] Key type .llcrypt unregistered [ 1282.411565] Key type ._llcrypt unregistered [ 1310.362359] Key type ._llcrypt registered [ 1310.366136] Key type .llcrypt registered [ 1310.496357] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 1312.437526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1312.514871] alg: No test for adler32 (adler32-zlib) [ 1313.813535] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 1314.196741] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 1315.927215] Key type lgssc registered [ 1317.236250] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1326.883791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1326.900126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1328.464354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1334.064500] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1338.850832] LustreError: 40053:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1338.867327] LustreError: 40053:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1343.632767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1344.090323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1349.791891] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1358.939734] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1366.206303] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1374.075201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1374.446855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1381.468973] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1389.925497] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1399.024646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1407.421917] Lustre: server umount lustre-MDT0000 complete [ 1409.509884] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1409.522331] LustreError: 41489:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1409.539772] Lustre: Skipped 1 previous similar message [ 1410.975651] LustreError: 40034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789064547 with bad export cookie 77214745181382351 [ 1410.990153] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1411.001290] LustreError: 40034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1412.279187] LustreError: 40048:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1412.336116] LustreError: 40048:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1412.347622] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.2@tcp (stopping) [ 1414.631621] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1414.646071] LustreError: 41489:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1414.651311] Lustre: Skipped 1 previous similar message [ 1414.659037] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1417.489703] Lustre: server umount lustre-MDT0001 complete [ 1440.650496] Lustre: server umount lustre-OST0000 complete [ 1459.818911] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 1462.706350] Key type lgssc unregistered [ 1463.116745] LNet: 43937:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1463.124384] LNetError: 43937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1463.135211] LNet: Removed LNI 192.168.202.102@tcp [ 1464.107409] Key type .llcrypt unregistered [ 1464.110046] Key type ._llcrypt unregistered [ 1494.650827] Key type ._llcrypt registered [ 1494.655930] Key type .llcrypt registered [ 1494.826817] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 1496.439698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1496.510133] alg: No test for adler32 (adler32-zlib) [ 1497.728844] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 1497.973458] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 1499.631294] Key type lgssc registered [ 1501.028260] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1509.395381] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1509.433198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1511.147204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1518.447183] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1521.648361] LustreError: 45380:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1521.681890] LustreError: 45380:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1526.761147] LustreError: 45381:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1531.486385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1531.875308] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1532.110341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1537.344237] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1544.659404] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1551.194226] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1559.734818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1560.120221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1567.564728] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1577.697398] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1581.612318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1588.931220] Lustre: lustre-OST0000: Not available for connect from 192.168.202.2@tcp (stopping) [ 1591.779574] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1591.787530] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1591.787707] LustreError: Skipped 1 previous similar message [ 1591.795364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1591.817943] Lustre: Skipped 1 previous similar message [ 1594.014767] Lustre: lustre-OST0000: Not available for connect from 192.168.202.2@tcp (stopping) [ 1594.033756] Lustre: Skipped 1 previous similar message [ 1594.760955] Lustre: server umount lustre-OST0000 complete [ 1613.794519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1613.812937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1613.829752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1616.355143] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1626.593478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1626.605084] Lustre: Skipped 5 previous similar messages [ 1628.127232] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1628.318165] Lustre: server umount lustre-MDT0000 complete [ 1631.712491] LustreError: 45376:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1631.719548] LustreError: 45376:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1631.898669] LustreError: 45362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789064768 with bad export cookie 10457324377368186712 [ 1631.900378] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1631.907569] LustreError: 45362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1638.678346] Lustre: server umount lustre-MDT0001 complete [ 1654.922929] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 1659.386623] Key type lgssc unregistered [ 1659.828578] LNet: 49093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1659.836868] LNetError: 49093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1659.869147] LNet: Removed LNI 192.168.202.102@tcp [ 1661.161842] Key type .llcrypt unregistered [ 1661.164386] Key type ._llcrypt unregistered [ 1690.835572] Key type ._llcrypt registered [ 1690.841405] Key type .llcrypt registered [ 1691.077474] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 1693.338527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1693.495966] alg: No test for adler32 (adler32-zlib) [ 1695.270606] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 1695.554231] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 1697.175205] Key type lgssc registered [ 1698.780292] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1710.960807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1710.988590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1712.369035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1718.480976] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1722.881397] LustreError: 50505:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1722.910670] LustreError: 50505:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1727.971135] LustreError: 50506:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1728.627921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1729.076570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1734.207091] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1741.788653] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1747.242446] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1754.098969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1754.443709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1757.561854] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1762.114103] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1773.796157] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1781.076700] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1784.956899] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1797.470233] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-1014h28m52s [ 1798.917978] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1801.908746] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.2@tcp (stopping) [ 1802.957974] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.2@tcp (stopping) [ 1803.757332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1803.788101] Lustre: Skipped 2 previous similar messages [ 1804.838226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1804.860901] Lustre: Skipped 5 previous similar messages [ 1807.676815] Lustre: server umount lustre-MDT0000 complete [ 1808.019524] LustreError: 50502:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1810.944094] LustreError: 51613:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1811.016033] LustreError: 51613:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1812.189076] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1817.074617] LustreError: 50501:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1817.117312] LustreError: 50501:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1823.009532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1823.230900] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1823.730070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1823.829845] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1829.713452] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1829.897524] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1834.335624] Lustre: DEBUG MARKER: Wait for df (36737) ... [ 1836.352819] Lustre: DEBUG MARKER: done [ 1844.193487] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1844.214058] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1844.248049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1844.705093] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1847.641425] Lustre: server umount lustre-OST0000 complete [ 1855.458712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1855.464827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1855.470040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1855.470055] Lustre: Skipped 3 previous similar messages [ 1855.512206] Lustre: Skipped 2 previous similar messages [ 1857.437232] Lustre: server umount lustre-MDT0000 complete [ 1862.210823] LustreError: 50486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789064999 with bad export cookie 16229832727734936911 [ 1862.211347] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1862.220091] LustreError: 50486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1862.838722] Lustre: server umount lustre-MDT0001 complete [ 1876.835386] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 1880.094278] Key type lgssc unregistered [ 1880.593050] LNet: 55434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1880.609763] LNetError: 55434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1880.632549] LNet: Removed LNI 192.168.202.102@tcp [ 1881.575356] Key type .llcrypt unregistered [ 1881.591170] Key type ._llcrypt unregistered [ 1927.799258] Key type ._llcrypt registered [ 1927.802672] Key type .llcrypt registered [ 1927.940350] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 1928.456625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1928.555897] alg: No test for adler32 (adler32-zlib) [ 1930.116690] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 1930.349636] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 1932.071203] Key type lgssc registered [ 1933.921393] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1945.552898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1945.566700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1947.082253] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1947.094083] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1947.189839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1952.479271] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1961.808684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1961.940468] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1961.976772] Lustre: 57638:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1962.242535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1966.841555] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1974.738786] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1981.437157] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1988.939053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1989.143280] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1989.336692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1996.268934] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1998.979335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2005.113792] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2012.340140] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-1014h32m27s [ 2014.442692] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2022.410761] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 2025.080633] Lustre: DEBUG MARKER: Creating a test file and stat it [ 2025.128532] LustreError: 57647:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 2045.617343] Lustre: lustre-MDT0000: Client b3e8ef84-6b72-45c5-8608-3797c12ee2ca (at 192.168.202.2@tcp) reconnecting [ 2065.207194] LustreError: 57647:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2068.088537] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2071.598427] Lustre: DEBUG MARKER: done [ 2080.737144] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2080.743222] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2080.763914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2081.250478] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2081.275569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2081.285604] Lustre: Skipped 1 previous similar message [ 2086.381436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2086.400314] Lustre: Skipped 1 previous similar message [ 2091.495065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2091.508683] Lustre: Skipped 1 previous similar message [ 2093.023175] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2093.416701] Lustre: server umount lustre-OST0000 complete [ 2100.704753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2100.718500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2100.738199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2104.344713] Lustre: server umount lustre-MDT0000 complete [ 2109.003591] LustreError: 57631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789065245 with bad export cookie 17296476644331400581 [ 2109.007252] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2109.013458] LustreError: 57631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2109.538277] Lustre: server umount lustre-MDT0001 complete [ 2121.798948] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2125.004790] Key type lgssc unregistered [ 2125.409834] LNet: 61971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2125.422894] LNetError: 61971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2125.440092] LNet: Removed LNI 192.168.202.102@tcp [ 2126.755878] Key type .llcrypt unregistered [ 2126.757719] Key type ._llcrypt unregistered [ 2191.452680] Key type ._llcrypt registered [ 2191.459037] Key type .llcrypt registered [ 2191.700598] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2194.512183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2194.546845] alg: No test for adler32 (adler32-zlib) [ 2196.293783] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 2196.882311] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 2198.727282] Key type lgssc registered [ 2199.747103] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2205.874104] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2205.898303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2207.436554] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2207.496111] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2207.813155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2209.382805] Lustre: Failing over lustre-MDT0000 [ 2209.962621] Lustre: server umount lustre-MDT0000 complete [ 2220.042316] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 14:35:55 (1789065355) [ 2229.484668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2230.761495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2239.026836] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2251.216464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2251.574775] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2251.656392] Lustre: 64774:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2252.389626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2260.150973] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2272.168737] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2280.564726] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2289.153125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2289.445150] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2289.819298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2298.709347] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2298.733864] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2309.107813] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2315.230431] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2324.456837] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2324.466588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2324.503224] Lustre: Skipped 1 previous similar message [ 2324.526694] Lustre: Skipped 2 previous similar messages [ 2329.590737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2329.607924] Lustre: Skipped 2 previous similar messages [ 2330.827805] Lustre: server umount lustre-MDT0000 complete [ 2335.412303] LustreError: 66631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789065472 with bad export cookie 18304377884725532051 [ 2335.420439] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2335.425687] LustreError: 66631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2335.875803] Lustre: server umount lustre-MDT0001 complete [ 2340.647642] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2347.186386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2355.190642] Lustre: 63693:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789065476/real 1789065476] req@0000000098383dfb x1875970970660992/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789065492 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2355.230497] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2355.244732] Lustre: Skipped 1 previous similar message [ 2356.191229] Lustre: 63691:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789065477/real 1789065477] req@000000009ddfe144 x1875970970661120/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789065493 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2359.538693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2360.351487] Lustre: 63691:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789065481/real 1789065481] req@00000000cd9b1445 x1875970970661248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789065497 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2360.975322] LustreError: 66625:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2361.005963] LustreError: 66625:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2361.126880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2361.238173] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2364.206174] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2364.212453] LustreError: 66626:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2364.239439] LustreError: 66626:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2365.219859] LustreError: 66625:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2365.246045] LustreError: 66625:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2367.119413] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2367.263470] Lustre: 63694:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789065488/real 1789065488] req@00000000bf189a15 x1875970970661632/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789065504 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2367.292016] Lustre: 63694:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2370.534439] LustreError: 68547:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2370.558543] LustreError: 68547:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2375.663957] LustreError: 68546:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2375.706542] LustreError: 68546:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2378.952201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2379.495271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2384.896888] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2386.768409] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2397.829883] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2405.636547] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2415.593852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2415.606989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2415.628576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2418.658834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2418.666607] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2418.687564] Lustre: Skipped 1 previous similar message [ 2418.716751] Lustre: Skipped 1 previous similar message [ 2421.665635] Lustre: server umount lustre-MDT0000 complete [ 2423.777722] LustreError: 68753:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2423.801141] LustreError: 68753:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2426.310767] LustreError: 64766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789065563 with bad export cookie 18304377884725534326 [ 2426.315409] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2426.323218] LustreError: 64766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2426.668633] Lustre: server umount lustre-MDT0001 complete [ 2434.251392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2445.029054] Lustre: 63694:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789065565/real 1789065565] req@0000000090ff5832 x1875970970693888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789065581 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2445.057103] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2446.431108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2451.231858] LustreError: 63690:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000078d0e529 x1875970970694912/t0(0) o250->MGC192.168.202.102@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2451.690542] LustreError: 66625:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2451.826400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2451.924968] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2457.951230] Lustre: 63693:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789065578/real 1789065578] req@000000007cb7f540 x1875970970694528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789065594 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2457.977665] Lustre: 63693:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2458.546802] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2470.328935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2473.971718] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2476.519922] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2489.388693] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2496.879604] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2506.742473] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2506.755741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2506.783485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2513.010498] Lustre: server umount lustre-MDT0000 complete [ 2514.914407] LustreError: 71237:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2514.925445] LustreError: 71237:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 2518.422465] LustreError: 64766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789065655 with bad export cookie 18304377884725535705 [ 2518.426301] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2518.442606] LustreError: 64766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2518.832300] Lustre: server umount lustre-MDT0001 complete [ 2527.427474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2532.293651] Lustre: DEBUG MARKER: files compared the same [ 2544.458748] Lustre: server umount lustre-OST0000 complete [ 2566.816840] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2571.293129] Key type lgssc unregistered [ 2571.732678] LNet: 74515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2571.746856] LNetError: 74515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2572.780156] LNet: Removed LNI 192.168.202.102@tcp [ 2574.259049] Key type .llcrypt unregistered [ 2574.260995] Key type ._llcrypt unregistered [ 2608.118920] Key type ._llcrypt registered [ 2608.121478] Key type .llcrypt registered [ 2608.251741] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2609.596613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2609.640815] alg: No test for adler32 (adler32-zlib) [ 2610.905979] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 2611.266706] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 2612.959375] Key type lgssc registered [ 2614.518902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2630.552459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2630.593897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2632.382664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2640.029082] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug +malloc all [ 2642.917554] LustreError: 75932:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2642.935150] LustreError: 75932:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2648.032964] LustreError: 75931:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2653.153589] LustreError: 75932:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2653.538514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2654.342257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2661.309226] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug +malloc all [ 2672.615279] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2679.950227] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2689.630465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2690.002425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2698.444183] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug +malloc all [ 2702.311073] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2710.644083] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2724.833689] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2724.860766] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2724.879196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2727.916234] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2727.917870] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2727.933700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2730.135845] Lustre: server umount lustre-OST0000 complete [ 2736.098741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2736.114272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2736.148497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2736.164220] Lustre: Skipped 1 previous similar message [ 2738.148673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2738.151638] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2742.071802] Lustre: server umount lustre-MDT0000 complete [ 2743.273686] LustreError: 75925:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2743.299690] LustreError: 75925:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2746.517116] LustreError: 75913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789065883 with bad export cookie 14741807342153313190 [ 2746.539920] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2746.566555] LustreError: 75913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2747.414672] Lustre: server umount lustre-MDT0001 complete [ 2764.962290] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2769.210459] Key type lgssc unregistered [ 2769.705414] LNet: 79501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2769.719784] LNetError: 79501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2769.745379] LNet: Removed LNI 192.168.202.102@tcp [ 2771.560375] Key type .llcrypt unregistered [ 2771.565599] Key type ._llcrypt unregistered [ 2803.627965] Key type ._llcrypt registered [ 2803.636208] Key type .llcrypt registered [ 2803.838497] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2805.548992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2805.694390] alg: No test for adler32 (adler32-zlib) [ 2807.172538] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 2807.704312] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 2809.527297] Key type lgssc registered [ 2811.434933] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2822.014650] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2822.035787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2837.663455] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2842.783427] LustreError: 80891:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.102@tcp: failed processing log, type 4: rc = -110 [ 2869.471207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2877.738799] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2887.979450] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2897.638232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2897.976555] Lustre: *** cfs_fail_loc=706, val=0*** [ 2898.503932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2898.624314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2903.527656] LustreError: 80916:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2903.541752] LustreError: 80916:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2903.715351] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2904.553177] LustreError: 80917:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2904.592883] LustreError: 80917:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2909.666625] LustreError: 81979:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2909.689457] LustreError: 81979:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2914.796883] LustreError: 81979:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2914.827506] LustreError: 81979:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2915.240130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2915.765637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2923.164109] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2935.432844] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2942.091257] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2946.540183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2946.541138] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2946.564721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2951.651137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2951.658474] Lustre: Skipped 1 previous similar message [ 2956.786128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2956.797442] Lustre: Skipped 1 previous similar message [ 2959.327234] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2959.555067] Lustre: server umount lustre-OST0000 complete [ 2967.009676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2967.010144] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2967.029093] Lustre: Skipped 1 previous similar message [ 2967.031745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2967.072051] Lustre: Skipped 2 previous similar messages [ 2971.475759] Lustre: server umount lustre-MDT0000 complete [ 2972.145508] LustreError: 81978:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2972.170965] LustreError: 81978:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2976.752172] LustreError: 80899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789066113 with bad export cookie 516559062614608653 [ 2976.768215] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2977.303190] Lustre: server umount lustre-MDT0001 complete [ 2992.274714] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2996.037750] Key type lgssc unregistered [ 2996.430873] LNet: 84510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2996.445378] LNetError: 84510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2996.491855] LNet: Removed LNI 192.168.202.102@tcp [ 2998.145956] Key type .llcrypt unregistered [ 2998.155554] Key type ._llcrypt unregistered [ 3030.136994] Key type ._llcrypt registered [ 3030.142203] Key type .llcrypt registered [ 3030.347672] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3032.850182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3032.909464] alg: No test for adler32 (adler32-zlib) [ 3034.188703] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 3034.739734] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 3036.607923] Key type lgssc registered [ 3038.174112] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3049.827191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3049.853694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3056.752644] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3065.774526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3066.417767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3071.460222] LustreError: 86621:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3071.482024] LustreError: 86621:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3071.612722] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3076.578240] LustreError: 86625:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3076.622300] LustreError: 86625:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3081.707133] LustreError: 86621:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3081.715653] LustreError: 86621:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3084.672653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3085.169052] LustreError: 86624:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3085.189305] LustreError: 86624:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3085.207581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3090.405061] LustreError: 86625:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3090.432607] LustreError: 86625:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3092.025925] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3101.684753] LustreError: 86621:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3101.707311] LustreError: 86621:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 3101.902089] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3101.920047] Lustre: MGS: Client aad21c17-abdc-458b-8515-b5ec774285d0 (at 0@lo) reconnecting [ 3101.942900] LustreError: 86635:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3102.254423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3102.451427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 3110.890120] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3127.777961] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3127.781739] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3127.792539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3127.814784] Lustre: Skipped 1 previous similar message [ 3132.906715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3132.928742] Lustre: Skipped 1 previous similar message [ 3134.047684] Lustre: server umount lustre-OST0000 complete [ 3143.136625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3143.140728] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3143.160176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3143.182495] Lustre: Skipped 1 previous similar message [ 3143.194225] Lustre: Skipped 2 previous similar messages [ 3145.004872] Lustre: server umount lustre-MDT0000 complete [ 3148.263718] LustreError: 88306:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3148.294734] LustreError: 88306:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3150.488589] Lustre: server umount lustre-MDT0001 complete [ 3154.627865] Lustre: server umount MGS complete [ 3169.282975] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 14:51:44 (1789066304) [ 3202.711942] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 3214.400185] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3256.203656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3261.092504] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3271.994718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3272.303232] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3272.365579] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3272.475218] Lustre: lustre-MDT0001: new disk, initializing [ 3272.606774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3272.661993] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3272.674904] Lustre: 93816:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3272.685820] LustreError: 93816:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3277.728505] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3288.760113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3289.087476] Lustre: lustre-OST0000: new disk, initializing [ 3289.098618] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3289.112734] Lustre: 94582:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3289.218076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3296.615628] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3298.848200] Lustre: 95007:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000b5f333d4 x1875971849367296/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789066451 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3305.889841] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3312.687559] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3312.701928] Lustre: MGS: Client b6ed8732-a8aa-410a-a5b8-38cb569ddcb5 (at 0@lo) reconnecting [ 3312.745794] Lustre: 95628:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3312.765931] LustreError: 95628:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 3312.781690] LustreError: 95628:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 3312.799562] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3312.873297] Lustre: lustre-MDT0000: new disk, initializing [ 3313.034952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3313.057785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3313.126969] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3314.161089] Lustre: 95652:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000005e383cf2 x1875971849374464/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789066467 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3318.374827] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3319.267545] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3323.245437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3323.253875] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3323.341443] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3348.961620] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3348.973481] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3349.001211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3351.015887] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3355.083631] Lustre: server umount lustre-OST0000 complete [ 3361.254513] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3361.258150] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3361.280684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3361.280694] Lustre: Skipped 2 previous similar messages [ 3365.689488] Lustre: server umount lustre-MDT0000 complete [ 3366.372837] LustreError: 96079:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3366.384242] LustreError: 96079:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3384.287207] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3384.746610] Lustre: server umount lustre-MDT0001 complete [ 3388.262628] Lustre: server umount MGS complete [ 3399.955456] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 14:55:35 (1789066535) [ 3418.580872] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 3421.664822] Key type lgssc unregistered [ 3421.932540] LNet: 98151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3421.938936] LNetError: 98151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3423.013809] LNet: Removed LNI 192.168.202.102@tcp [ 3423.794152] Key type .llcrypt unregistered [ 3423.799147] Key type ._llcrypt unregistered [ 3441.052436] Key type ._llcrypt registered [ 3441.055034] Key type .llcrypt registered [ 3441.190695] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3442.418490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3442.474055] alg: No test for adler32 (adler32-zlib) [ 3443.516265] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 3443.885080] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 3445.575298] Key type lgssc registered [ 3447.002420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3454.403865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3454.428445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3456.001495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3461.447487] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3465.184175] LustreError: 99370:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3465.202748] LustreError: 99370:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3470.315514] LustreError: 99371:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3470.921155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3471.474393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3477.548741] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3487.056739] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3494.191492] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3501.751814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3502.081554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3508.149782] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3516.837639] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3525.639144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3535.840500] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3535.855913] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3535.865287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3535.891232] Lustre: Skipped 1 previous similar message [ 3540.961180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3540.969529] Lustre: Skipped 2 previous similar messages [ 3541.561537] Lustre: server umount lustre-OST0000 complete [ 3546.085289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3546.088260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3546.098701] Lustre: Skipped 1 previous similar message [ 3546.107925] Lustre: Skipped 1 previous similar message [ 3551.204514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3551.214633] Lustre: Skipped 1 previous similar message [ 3551.838149] Lustre: server umount lustre-MDT0000 complete [ 3556.010315] LustreError: 99352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789066693 with bad export cookie 1674505549056512793 [ 3556.011269] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3556.018018] LustreError: 99352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3556.450709] Lustre: server umount lustre-MDT0001 complete [ 3567.826412] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 3570.589627] Key type lgssc unregistered [ 3570.862555] LNet: 102925:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3570.875931] LNetError: 102925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3570.890302] LNet: Removed LNI 192.168.202.102@tcp [ 3571.837531] Key type .llcrypt unregistered [ 3571.839827] Key type ._llcrypt unregistered [ 3589.052985] Key type ._llcrypt registered [ 3589.057218] Key type .llcrypt registered [ 3589.173594] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3590.339027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3590.462391] alg: No test for adler32 (adler32-zlib) [ 3591.485699] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 3591.648170] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 3593.344964] Key type lgssc registered [ 3594.298974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3599.676882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3599.697349] LustreError: 104001:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3599.740925] LustreError: 104005:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3599.747489] LustreError: 104001:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4963 [ 3599.755342] LustreError: 104001:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3599.773474] LustreError: 104001:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3599.805065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3601.367564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3609.598518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3609.873336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3614.969714] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3621.297735] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing lsmod [ 3624.324746] LustreError: 105207:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3624.362481] LustreError: 105211:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3624.368817] LustreError: 105207:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4967 [ 3624.372106] LustreError: 105207:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3624.382393] LustreError: 105207:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3624.389771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3624.885213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3626.958440] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3630.561886] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3630.570232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3630.598087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3632.096785] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3632.097385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3632.131076] Lustre: Skipped 1 previous similar message [ 3632.157085] Lustre: Skipped 2 previous similar messages [ 3634.214063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3634.223725] Lustre: Skipped 2 previous similar messages [ 3634.776807] Lustre: server umount lustre-MDT0000 complete [ 3639.299520] LustreError: 104042:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3639.331365] LustreError: 104042:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3639.566847] LustreError: 104020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789066776 with bad export cookie 14547993314746386255 [ 3639.574859] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3639.585599] LustreError: 104020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3639.604248] Lustre: 104034:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.102@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3640.415662] LustreError: 104472:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000d78b25f1 x1875972433401472/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3640.668237] Lustre: server umount lustre-MDT0001 complete [ 3655.164804] Lustre: server umount lustre-OST0000 complete [ 3665.393131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3666.119973] LustreError: 106275:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3666.144982] LustreError: 106275:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3666.308283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3671.523672] LustreError: 106276:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3671.847410] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3676.643216] LustreError: 106275:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3680.478158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3680.847692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3685.974748] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3694.237569] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3700.433241] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3707.967612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3708.307318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3711.453062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3715.394084] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3723.327849] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3736.035260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3736.046314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3736.053348] Lustre: Skipped 1 previous similar message [ 3736.075034] Lustre: Skipped 1 previous similar message [ 3740.802326] Lustre: server umount lustre-OST0000 complete [ 3746.273863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3746.295040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3746.302481] Lustre: Skipped 1 previous similar message [ 3751.059523] Lustre: server umount lustre-MDT0000 complete [ 3751.392744] LustreError: 108114:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3751.412744] LustreError: 108114:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3755.109190] LustreError: 106257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789066892 with bad export cookie 14547993314746387172 [ 3755.119983] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3755.126107] LustreError: 106257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3755.882780] Lustre: server umount lustre-MDT0001 complete [ 3767.961936] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 3771.393845] Key type lgssc unregistered [ 3771.743742] LNet: 109820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3771.752184] LNetError: 109820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3771.781181] LNet: Removed LNI 192.168.202.102@tcp [ 3772.934693] Key type .llcrypt unregistered [ 3772.939151] Key type ._llcrypt unregistered [ 3799.199778] Key type ._llcrypt registered [ 3799.202987] Key type .llcrypt registered [ 3799.318432] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3800.926022] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3801.115762] alg: No test for adler32 (adler32-zlib) [ 3802.538276] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 3802.954598] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 3804.807317] Key type lgssc registered [ 3806.415372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3816.470564] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3816.497786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3817.950334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3824.229227] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3827.181000] LustreError: 111234:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3827.207062] LustreError: 111234:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3832.290172] LustreError: 111235:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3835.017766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3835.253540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3840.796350] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3848.647975] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3855.547438] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3862.786493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3863.231796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3870.226179] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3878.109688] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3887.800344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3893.784176] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3897.366038] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3903.460838] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3903.482917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3903.491269] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3908.578409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3908.586951] Lustre: Skipped 2 previous similar messages [ 3913.702022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3913.710059] Lustre: Skipped 1 previous similar message [ 3916.255171] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3916.417894] Lustre: server umount lustre-OST0000 complete [ 3922.404319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3922.417726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3922.449089] Lustre: Skipped 1 previous similar message [ 3922.461176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3923.945306] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3926.808321] Lustre: server umount lustre-MDT0000 complete [ 3929.056320] LustreError: 111230:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3929.092205] LustreError: 111230:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3931.668663] LustreError: 111214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789067068 with bad export cookie 14401290836379852888 [ 3931.675804] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3931.686301] LustreError: 111214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3932.201960] Lustre: server umount lustre-MDT0001 complete [ 3945.434964] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 3947.770752] Key type lgssc unregistered [ 3948.103889] LNet: 114896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3948.120433] LNetError: 114896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3948.157900] LNet: Removed LNI 192.168.202.102@tcp [ 3948.946382] Key type .llcrypt unregistered [ 3948.947534] Key type ._llcrypt unregistered [ 3969.751260] Key type ._llcrypt registered [ 3969.753762] Key type .llcrypt registered [ 3969.919267] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3971.883615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3971.934578] alg: No test for adler32 (adler32-zlib) [ 3972.932853] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 3973.186645] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 3974.871275] Key type lgssc registered [ 3976.203752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3984.717790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3984.740305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3986.244373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3991.810224] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3996.647527] LustreError: 116119:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3996.657836] LustreError: 116119:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4000.236217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4000.510855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4006.143891] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4016.728729] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4024.605542] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4032.711793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4033.036288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4038.852725] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4048.056588] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4057.647408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4067.831392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4067.831392] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4067.831940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4067.838229] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4067.849368] Lustre: Skipped 1 previous similar message [ 4072.187420] Lustre: server umount lustre-OST0000 complete [ 4077.547177] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4077.565265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4077.586153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4081.898164] Lustre: server umount lustre-MDT0000 complete [ 4083.173246] LustreError: 116119:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4083.192945] LustreError: 116119:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4085.502453] LustreError: 116099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789067222 with bad export cookie 4364713255665037499 [ 4085.517040] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4085.524167] LustreError: 116099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4086.052227] Lustre: server umount lustre-MDT0001 complete [ 4099.222798] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 4102.947864] Key type lgssc unregistered [ 4103.311800] LNet: 119676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4103.323781] LNetError: 119676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4103.354583] LNet: Removed LNI 192.168.202.102@tcp [ 4104.537443] Key type .llcrypt unregistered [ 4104.544323] Key type ._llcrypt unregistered [ 4133.057017] Key type ._llcrypt registered [ 4133.060866] Key type .llcrypt registered [ 4133.173585] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 4134.068433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4134.123418] alg: No test for adler32 (adler32-zlib) [ 4135.126966] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 4135.287763] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 4136.920468] Key type lgssc registered [ 4138.078812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4146.560840] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4146.598565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4148.556212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4154.246493] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4158.949857] LustreError: 121090:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4158.981056] LustreError: 121090:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4164.070960] LustreError: 121091:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4164.297566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4164.871316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4170.734346] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4179.905331] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4186.141862] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4193.529618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4193.699771] LustreError: 122912:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4193.891748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4198.913627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 4200.988317] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4209.109455] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4216.185550] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4219.870531] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4219.877393] Lustre: Skipped 2 previous similar messages [ 4223.672146] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4223.677600] Lustre: Skipped 2 previous similar messages [ 4225.628364] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4225.670941] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4229.961415] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.2@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4229.967569] Lustre: Skipped 2 previous similar messages [ 4233.374097] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.2@tcp 192.168.0.[2,10]@tcp [ 4240.621221] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.2/24@tcp in log lustre-MDT0000 [ 4240.629113] Lustre: Skipped 2 previous similar messages [ 4243.553726] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.2/24@tcp [ 4243.569012] Lustre: Skipped 1 previous similar message [ 4250.098557] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4250.102890] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4250.117435] LustreError: Skipped 1 previous similar message [ 4250.119354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4250.126024] Lustre: Skipped 1 previous similar message [ 4255.212175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4255.228400] Lustre: Skipped 1 previous similar message [ 4255.983535] Lustre: server umount lustre-OST0000 complete [ 4262.372093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4262.383259] LustreError: Skipped 1 previous similar message [ 4262.387814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4262.404026] Lustre: Skipped 1 previous similar message [ 4262.411481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4265.440634] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4265.442339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4265.476401] Lustre: Skipped 1 previous similar message [ 4266.819262] Lustre: server umount lustre-MDT0000 complete [ 4270.560551] LustreError: 121090:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4270.598505] LustreError: 121090:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4271.188725] LustreError: 124371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789067408 with bad export cookie 3348719783252970517 [ 4271.190939] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4271.200323] LustreError: 124371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4271.664885] Lustre: server umount lustre-MDT0001 complete [ 4283.882281] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 4287.337850] Key type lgssc unregistered [ 4287.706078] LNet: 125833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4287.717484] LNetError: 125833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4287.738305] LNet: Removed LNI 192.168.202.102@tcp [ 4288.735725] Key type .llcrypt unregistered [ 4288.737428] Key type ._llcrypt unregistered [ 4317.345991] Key type ._llcrypt registered [ 4317.351548] Key type .llcrypt registered [ 4317.468543] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 4318.570043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4318.762800] alg: No test for adler32 (adler32-zlib) [ 4319.775291] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 4319.942232] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 4321.583411] Key type lgssc registered [ 4322.774759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4332.700825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4332.730972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4334.199605] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4334.220331] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.2/24@tcp [ 4334.257661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4339.764804] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4344.809041] LustreError: 127391:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4344.833929] LustreError: 127391:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4349.207518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4349.599176] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4349.623462] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.2/24@tcp [ 4349.689016] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4354.436585] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4362.026334] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4369.138135] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4377.512051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4377.913136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4379.985257] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 4384.830807] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4393.909366] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4409.825353] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4409.827252] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4409.830073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4409.875663] Lustre: Skipped 1 previous similar message [ 4412.210482] Lustre: server umount lustre-OST0000 complete [ 4420.067988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4420.069859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4420.083493] Lustre: Skipped 1 previous similar message [ 4420.103608] Lustre: Skipped 2 previous similar messages [ 4423.240764] Lustre: server umount lustre-MDT0000 complete [ 4425.187443] LustreError: 127390:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4425.220948] LustreError: 127390:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4427.595429] LustreError: 127372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789067564 with bad export cookie 16789245039667681107 [ 4427.601079] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4427.610118] LustreError: 127372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4428.206178] Lustre: server umount lustre-MDT0001 complete [ 4442.036059] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 4445.417876] Key type lgssc unregistered [ 4445.880752] LNet: 130980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4445.897603] LNetError: 130980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4445.912988] LNet: Removed LNI 192.168.202.102@tcp [ 4446.798902] Key type .llcrypt unregistered [ 4446.801473] Key type ._llcrypt unregistered [ 4476.068724] Key type ._llcrypt registered [ 4476.070987] Key type .llcrypt registered [ 4476.188738] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 4495.079421] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 4495.898430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4495.924398] alg: No test for adler32 (adler32-zlib) [ 4497.104850] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 4497.365671] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 4499.087693] Key type lgssc registered [ 4500.907628] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4541.443387] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4541.498942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4542.875665] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4542.987352] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4543.325103] Lustre: lustre-MDT0000: new disk, initializing [ 4543.581108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4543.632961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4548.658618] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4560.666017] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4560.775781] Lustre: 134383:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4560.855991] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4560.864488] Lustre: Skipped 1 previous similar message [ 4561.023985] Lustre: lustre-MDT0001: new disk, initializing [ 4561.217099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4561.292940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4561.305121] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4567.798673] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4578.668189] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4584.086840] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4590.948509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4591.189849] Lustre: lustre-OST0000: new disk, initializing [ 4591.192854] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4591.197634] Lustre: 136351:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4591.274823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4597.784819] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4601.408374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4601.422957] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4601.600028] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4608.649837] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4613.420607] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4613.622146] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4618.320724] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4618.492263] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4622.316235] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4622.322699] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4622.323071] LustreError: Skipped 1 previous similar message [ 4622.326252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4622.342790] Lustre: Skipped 1 previous similar message [ 4626.672611] Lustre: server umount lustre-OST0000 complete [ 4632.555745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4632.580615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4632.583347] Lustre: Skipped 1 previous similar message [ 4637.666382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4637.673577] Lustre: Skipped 1 previous similar message [ 4642.786361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4642.792372] Lustre: Skipped 1 previous similar message [ 4645.343126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4645.613553] Lustre: server umount lustre-MDT0000 complete [ 4648.647607] LustreError: 134375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789067785 with bad export cookie 16942089958339824305 [ 4648.651064] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4648.661761] LustreError: 134375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4648.989979] Lustre: server umount lustre-MDT0001 complete [ 4659.785797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4660.166825] LustreError: 138570:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4660.193071] LustreError: 138570:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4660.281911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4665.315052] LustreError: 138571:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4665.377457] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4670.433140] LustreError: 138570:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4674.210609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4674.434564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4679.250482] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4685.732713] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4691.470270] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4698.548437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4698.859866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4705.222940] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4712.896607] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4717.735577] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4718.097054] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4723.376885] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4723.666862] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4733.582624] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4733.761532] Lustre: lustre-OST0001: new disk, initializing [ 4733.770084] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4733.779394] Lustre: 141941:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4733.896515] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4739.952140] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4741.284434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4741.304270] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4741.415755] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4748.860844] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4749.022947] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4753.409480] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4753.603305] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4758.926866] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8cd8471f2000.ost_server_uuid 50 [ 4760.311731] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8cd8471f2000.ost_server_uuid in FULL state after 0 sec [ 4770.537875] Lustre: server umount lustre-OST0001 complete [ 4772.322845] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4772.329453] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4772.343303] LustreError: 140391:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4772.345996] Lustre: Skipped 2 previous similar messages [ 4772.358821] LustreError: 140391:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4776.940778] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4776.958078] LustreError: Skipped 1 previous similar message [ 4776.965916] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4776.989272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4777.460649] LustreError: 140393:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4777.503735] LustreError: 140393:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4780.760870] Lustre: server umount lustre-OST0000 complete [ 4787.168925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4787.177240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4787.188889] Lustre: Skipped 1 previous similar message [ 4787.196305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4787.198081] Lustre: Skipped 2 previous similar messages [ 4790.968889] Lustre: server umount lustre-MDT0000 complete [ 4792.801064] LustreError: 138571:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4792.819924] LustreError: 138571:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4795.039145] LustreError: 138550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789067932 with bad export cookie 16942089958339825187 [ 4795.045038] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4795.049644] LustreError: 138550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4795.272767] Lustre: server umount lustre-MDT0001 complete [ 4815.413686] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 4818.880433] Key type lgssc unregistered [ 4819.208350] LNet: 144944:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4819.216887] LNetError: 144944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4819.230958] LNet: Removed LNI 192.168.202.102@tcp [ 4820.156927] Key type .llcrypt unregistered [ 4820.160903] Key type ._llcrypt unregistered [ 4864.872402] Key type ._llcrypt registered [ 4864.875110] Key type .llcrypt registered [ 4865.048727] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 4881.562762] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 4883.611295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4883.654717] alg: No test for adler32 (adler32-zlib) [ 4885.001807] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 4885.574445] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 4887.464351] Key type lgssc registered [ 4889.147820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4927.078584] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4927.095776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4928.361823] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4928.410980] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4928.520479] Lustre: lustre-MDT0000: new disk, initializing [ 4928.676147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4928.706527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4933.951888] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4945.302951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4945.514375] Lustre: 149095:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4945.608230] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4945.614777] Lustre: Skipped 1 previous similar message [ 4945.734347] Lustre: lustre-MDT0001: new disk, initializing [ 4945.889635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4945.933793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4945.945127] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4951.074416] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4961.608110] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4967.550598] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4973.898741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4974.223557] Lustre: lustre-OST0000: new disk, initializing [ 4974.226833] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4974.237677] Lustre: 151063:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4974.371425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4980.918684] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4984.427636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4984.447655] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4984.597357] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4992.073718] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4997.065729] Lustre: Failing over lustre-OST0000 [ 4997.305784] Lustre: server umount lustre-OST0000 complete [ 5015.520755] Lustre: 147317:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789068136/real 1789068136] req@00000000f7c8286b x1875973790270208/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1789068152 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5015.561680] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5016.553102] Lustre: 147319:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789068137/real 1789068137] req@00000000ff617d69 x1875973790270464/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789068153 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5016.607266] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5017.329523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5017.702456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5017.733284] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 5021.733992] Lustre: 147317:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789068142/real 1789068142] req@00000000b4707a4d x1875973790272256/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789068158 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5021.768871] Lustre: 147317:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 5024.434772] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5025.434202] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 5026.847287] Lustre: 147317:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789068147/real 1789068147] req@000000000d5f78bd x1875973790273408/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789068163 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5026.951568] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5026.954724] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 5028.529217] Lustre: Failing over lustre-MDT0000 [ 5028.823363] Lustre: server umount lustre-MDT0000 complete [ 5030.948148] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5032.032143] Lustre: 147318:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789068152/real 1789068152] req@0000000066d5302e x1875973790274432/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789068168 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5032.075802] Lustre: 147318:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5035.670324] LustreError: 150327:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5035.680864] LustreError: 150327:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 5040.781514] LustreError: 149103:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5040.796106] LustreError: 149103:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5045.909653] LustreError: 149104:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5045.937406] LustreError: 149104:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5046.239323] Lustre: 147318:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789068167/real 1789068167] req@0000000088549423 x1875973790285696/t0(0) o400->MGC192.168.202.102@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789068183 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5046.269724] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5050.504794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5051.086310] LustreError: 149104:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5051.115863] LustreError: 149104:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 5056.484602] LustreError: 150327:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5056.489552] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7587cd95136a4531 [ 5056.522835] LustreError: 150327:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 5056.571227] Lustre: MGC192.168.202.102@tcp: Connection restored to 0@lo (at 0@lo) [ 5056.578892] Lustre: Skipped 1 previous similar message [ 5056.871689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5056.940253] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5057.165042] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5062.122400] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5062.148367] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5062.189879] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5062.594332] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5072.370875] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5072.376447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5072.379949] Lustre: Skipped 3 previous similar messages [ 5072.392937] Lustre: Skipped 1 previous similar message [ 5077.583470] Lustre: server umount lustre-OST0000 complete [ 5082.592476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5082.603541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5082.615563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5082.620461] Lustre: Skipped 1 previous similar message [ 5082.641105] Lustre: Skipped 1 previous similar message [ 5087.714411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5087.728637] Lustre: Skipped 1 previous similar message [ 5088.597518] Lustre: server umount lustre-MDT0000 complete [ 5092.458209] LustreError: 151062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789068229 with bad export cookie 8468963664453059889 [ 5092.471144] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5092.471615] LustreError: 151062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5092.952924] Lustre: server umount lustre-MDT0001 complete [ 5104.907752] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 5108.075830] Key type lgssc unregistered [ 5108.294642] LNet: 154720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5108.299904] LNetError: 154720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5108.315345] LNet: Removed LNI 192.168.202.102@tcp [ 5109.338653] Key type .llcrypt unregistered [ 5109.340605] Key type ._llcrypt unregistered [ 5153.796764] Key type ._llcrypt registered [ 5153.802205] Key type .llcrypt registered [ 5153.966687] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 5156.174592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5156.250834] alg: No test for adler32 (adler32-zlib) [ 5158.176447] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 5158.718487] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 5160.609916] Key type lgssc registered [ 5162.900967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5172.031667] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5172.041246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5173.643464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5178.865365] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5183.979631] LustreError: 156253:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5184.006225] LustreError: 156253:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5188.590872] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5189.088712] LustreError: 156254:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5192.889454] LustreError: 156282:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5192.898725] LustreError: 156282:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5192.907643] LustreError: 156282:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 5193.298116] Lustre: server umount lustre-MDT0000 complete [ 5210.359750] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 5219.088130] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 5260.091660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5260.316981] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5260.339519] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5260.444296] Lustre: lustre-MDT0000: new disk, initializing [ 5260.525035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5260.541780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5265.316592] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5277.698984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5277.886517] Lustre: 159590:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5277.948854] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5277.952635] Lustre: Skipped 1 previous similar message [ 5278.054318] Lustre: lustre-MDT0001: new disk, initializing [ 5278.177475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5278.262510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5278.276351] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5282.890466] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5293.560917] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5299.726795] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5306.216159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5306.558777] Lustre: lustre-OST0000: new disk, initializing [ 5306.563479] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5306.571201] Lustre: 161558:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5306.716679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5313.462554] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5313.655392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5313.676220] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5313.758335] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5322.215533] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5326.928965] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5328.785169] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 15:27:44 (1789068464) [ 5549.536414] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5549.537455] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5549.550707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5549.586841] Lustre: Skipped 1 previous similar message [ 5551.998267] Lustre: server umount lustre-MDT0000 complete [ 5554.655905] LustreError: 159600:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5554.669150] LustreError: 159600:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5555.668308] LustreError: 159581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789068692 with bad export cookie 16630475298600707906 [ 5555.672134] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5555.675275] LustreError: 159581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5559.775733] LustreError: 159597:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5559.776697] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5559.811940] Lustre: Skipped 1 previous similar message [ 5559.824660] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5559.832721] Lustre: Skipped 2 previous similar messages [ 5562.202982] Lustre: server umount lustre-MDT0001 complete [ 5572.271963] Lustre: server umount lustre-OST0000 complete [ 5581.912792] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 5593.286143] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 5629.263872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5629.439836] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5629.463237] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5629.551181] Lustre: lustre-MDT0000: new disk, initializing [ 5629.661463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5629.684528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5634.762201] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5645.734260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5645.964554] Lustre: 166214:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5646.010841] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5646.014385] Lustre: Skipped 1 previous similar message [ 5646.061237] Lustre: lustre-MDT0001: new disk, initializing [ 5646.111343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5646.153943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5646.170868] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5650.082163] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5660.241370] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5666.474219] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5673.244692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5673.612735] Lustre: lustre-OST0000: new disk, initializing [ 5673.623644] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5673.638305] Lustre: 168184:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5673.746869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5680.122758] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5682.223404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5682.242198] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5682.317893] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5689.124994] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5694.667679] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5694.977121] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5700.168409] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5700.458377] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5703.145593] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5703.155138] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5703.163955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5703.163964] Lustre: Skipped 1 previous similar message [ 5703.172409] Lustre: Skipped 1 previous similar message [ 5703.204132] LustreError: Skipped 1 previous similar message [ 5708.269074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5708.278955] Lustre: Skipped 2 previous similar messages [ 5708.672178] Lustre: server umount lustre-OST0000 complete [ 5713.376785] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5713.402292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5713.405453] Lustre: Skipped 1 previous similar message [ 5713.427628] Lustre: Skipped 1 previous similar message [ 5719.707448] Lustre: server umount lustre-MDT0000 complete [ 5723.622241] LustreError: 166227:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5723.648540] LustreError: 166227:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5723.736311] LustreError: 166208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789068860 with bad export cookie 16630475298600866736 [ 5723.738697] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5723.748838] LustreError: 166208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5724.026710] Lustre: server umount lustre-MDT0001 complete [ 5732.893587] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 15:34:28 (1789068868) [ 5756.250279] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 5767.057719] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 5809.685799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5809.937454] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5809.966670] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5810.063766] Lustre: lustre-MDT0000: new disk, initializing [ 5810.164232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5810.197776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5815.873067] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5827.940378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5828.169247] Lustre: 172942:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5828.618314] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5833.434644] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5843.188245] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5848.540343] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5856.049768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5856.238748] Lustre: lustre-OST0000: new disk, initializing [ 5856.240820] Lustre: Skipped 1 previous similar message [ 5856.243495] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5856.245760] Lustre: Skipped 2 previous similar messages [ 5856.248916] Lustre: 174918:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5862.988355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5862.998691] Lustre: Skipped 1 previous similar message [ 5863.008866] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5863.152396] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5864.532786] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5872.695780] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5883.873562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5883.882496] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5883.891698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5883.891706] Lustre: Skipped 2 previous similar messages [ 5885.838715] Lustre: server umount lustre-OST0000 complete [ 5890.016491] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5894.116047] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5894.124365] Lustre: Skipped 2 previous similar messages [ 5903.332248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5903.342498] Lustre: Skipped 7 previous similar messages [ 5904.351709] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5904.543815] Lustre: server umount lustre-MDT0000 complete [ 5908.456500] LustreError: 172952:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5908.484947] LustreError: 172952:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5908.749152] LustreError: 172935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789069045 with bad export cookie 16630475298600867618 [ 5908.756138] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5908.761530] LustreError: 172935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5909.078659] Lustre: server umount lustre-MDT0001 complete [ 5917.976547] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 15:37:33 (1789069053) [ 5937.415693] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 5950.134340] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 5989.612753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5989.943959] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5989.999403] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5990.135450] Lustre: lustre-MDT0000: new disk, initializing [ 5990.293459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5990.306524] Lustre: Skipped 2 previous similar messages [ 5990.337233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5995.713441] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6007.397264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6007.553827] Lustre: 179247:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6007.830978] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6012.944565] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6023.370529] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6029.691860] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6037.424281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6037.832543] Lustre: 181226:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6045.805545] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6045.959067] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6046.024392] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6058.286967] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6066.659334] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6066.659401] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6066.660220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6066.663937] LustreError: Skipped 1 previous similar message [ 6066.677019] Lustre: Skipped 1 previous similar message [ 6073.548086] Lustre: server umount lustre-OST0000 complete [ 6087.088833] Lustre: server umount lustre-MDT0000 complete [ 6087.142871] LustreError: 180130:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6087.188149] LustreError: 180130:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6092.208619] LustreError: 179238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789069229 with bad export cookie 16630475298600868703 [ 6092.219395] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6092.227650] LustreError: 179238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6092.865861] Lustre: server umount lustre-MDT0001 complete [ 6107.564837] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 6110.647278] Key type lgssc unregistered [ 6110.967658] LNet: 183082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6110.979530] LNetError: 183082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6111.000171] LNet: Removed LNI 192.168.202.102@tcp [ 6111.937120] Key type .llcrypt unregistered [ 6111.939701] Key type ._llcrypt unregistered [ 6141.884769] Key type ._llcrypt registered [ 6141.896527] Key type .llcrypt registered [ 6142.009442] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 6143.338216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6143.382690] alg: No test for adler32 (adler32-zlib) [ 6144.314992] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 6144.445592] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 6146.210670] Key type lgssc registered [ 6147.483098] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6156.965703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6156.985813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6158.343884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6164.677074] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6168.562653] LustreError: 184496:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6168.600939] LustreError: 184496:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6173.671310] LustreError: 184497:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6174.361438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6174.653067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6180.631034] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6189.998648] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6197.517459] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6205.115352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6205.443853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6212.594423] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6221.043533] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6228.981618] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6249.441250] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6249.445408] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6249.447350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6249.470675] Lustre: Skipped 1 previous similar message [ 6254.579747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6254.586427] Lustre: Skipped 2 previous similar messages [ 6255.391722] Lustre: server umount lustre-OST0000 complete [ 6259.690565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6259.705066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6259.729732] Lustre: Skipped 1 previous similar message [ 6264.807101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6264.819864] Lustre: Skipped 2 previous similar messages [ 6265.282935] Lustre: server umount lustre-MDT0000 complete [ 6269.078847] LustreError: 184477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789069406 with bad export cookie 8895353862441008874 [ 6269.095176] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6269.103361] LustreError: 184477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6269.804392] Lustre: server umount lustre-MDT0001 complete [ 6282.009933] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 6284.303388] Key type lgssc unregistered [ 6284.678435] LNet: 188051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6284.690120] LNetError: 188051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6284.715698] LNet: Removed LNI 192.168.202.102@tcp [ 6285.542785] Key type .llcrypt unregistered [ 6285.544197] Key type ._llcrypt unregistered [ 6311.594633] Key type ._llcrypt registered [ 6311.598700] Key type .llcrypt registered [ 6311.811050] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 6314.243661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6314.821467] alg: No test for adler32 (adler32-zlib) [ 6315.889161] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 6316.063655] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 6317.719302] Key type lgssc registered [ 6319.949338] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6329.473532] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6329.502980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6331.192831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6336.423361] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6341.606836] LustreError: 189472:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6341.627256] LustreError: 189472:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6346.444717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6346.876564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6352.084698] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6360.854563] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6366.912167] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6373.348238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6373.586827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6376.696710] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6380.860268] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6388.197356] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6401.210622] Lustre: lustre-OST0000: Not available for connect from 192.168.202.2@tcp (stopping) [ 6401.507398] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6401.525949] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6406.284874] Lustre: lustre-OST0000: Not available for connect from 192.168.202.2@tcp (stopping) [ 6406.291833] Lustre: Skipped 2 previous similar messages [ 6411.405287] Lustre: lustre-OST0000: Not available for connect from 192.168.202.2@tcp (stopping) [ 6411.423134] Lustre: Skipped 2 previous similar messages [ 6412.255129] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6412.560689] Lustre: server umount lustre-OST0000 complete [ 6422.840471] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8cd847280800.ost_server_uuid 50 [ 6425.356623] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6438.880159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6438.888410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6438.904260] Lustre: Skipped 1 previous similar message [ 6438.919454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6438.922269] Lustre: Skipped 2 previous similar messages [ 6442.023060] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6444.314605] Lustre: server umount lustre-MDT0000 complete [ 6447.073268] LustreError: 189469:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6447.101177] LustreError: 189469:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6448.279883] LustreError: 189454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789069585 with bad export cookie 10363759440829352578 [ 6448.289299] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6448.291795] LustreError: 189454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6448.581523] Lustre: server umount lustre-MDT0001 complete [ 6456.841097] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 15:46:32 (1789069592) [ 6463.606965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6463.980049] LustreError: 193387:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6464.040770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6468.649761] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6469.091142] LustreError: 193388:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6469.122678] LustreError: 193388:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6473.202248] LustreError: 193387:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6477.118762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6477.537220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6482.619543] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6490.035990] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6495.726783] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6503.023739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6503.487352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6508.591090] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6510.614492] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6518.617779] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6526.036846] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6526.222903] Lustre: lustre-OST0001: new disk, initializing [ 6526.225175] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6526.228714] Lustre: 196228:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6526.299666] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6532.734308] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6533.750765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6533.761392] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6533.876251] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6542.399828] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6548.453475] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6548.475631] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6548.506737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6548.523175] Lustre: Skipped 2 previous similar messages [ 6558.867347] Lustre: lustre-OST0000: Not available for connect from 192.168.202.2@tcp (stopping) [ 6558.872555] Lustre: Skipped 6 previous similar messages [ 6560.223161] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6560.499389] Lustre: server umount lustre-OST0000 complete [ 6563.984249] LustreError: 195208:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6568.509469] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6568.859791] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6573.732923] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6573.867425] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6580.207763] LustreError: 196647:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6580.224889] LustreError: 196647:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 6590.442817] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6590.448413] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6590.469738] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6590.469754] Lustre: Skipped 2 previous similar messages [ 6590.477788] Lustre: Skipped 2 previous similar messages [ 6590.513458] LustreError: Skipped 2 previous similar messages [ 6593.858485] Lustre: server umount lustre-OST0001 complete [ 6600.675840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6600.679467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6605.680356] Lustre: server umount lustre-MDT0000 complete [ 6610.059731] LustreError: 193368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789069747 with bad export cookie 10363759440829353649 [ 6610.064407] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6610.075039] LustreError: 193368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6610.521173] Lustre: server umount lustre-MDT0001 complete [ 6635.876470] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 15:49:31 (1789069771) [ 6642.844751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6643.243594] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6643.327882] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6643.809623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6649.342716] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6657.994809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6658.063704] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6658.078328] Lustre: 199673:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6662.876105] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6673.119888] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6679.942140] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6687.518534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6687.774947] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6695.019145] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6697.153799] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6702.833602] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6710.265573] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6710.492418] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6710.760640] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6710.768113] Lustre: Skipped 2 previous similar messages [ 6717.236633] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6725.324371] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6728.171316] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6728.183832] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6728.184777] LustreError: Skipped 1 previous similar message [ 6728.185771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6728.185776] Lustre: Skipped 3 previous similar messages [ 6728.232994] Lustre: Skipped 2 previous similar messages [ 6734.397389] Lustre: server umount lustre-OST0000 complete [ 6736.525349] LustreError: 202181:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6736.555170] LustreError: 202181:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 6751.713752] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6756.614792] Lustre: server umount lustre-OST0001 complete [ 6760.931518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6760.940662] Lustre: Skipped 2 previous similar messages [ 6766.830305] Lustre: server umount lustre-MDT0000 complete [ 6770.552458] LustreError: 199665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789069907 with bad export cookie 10363759440829355056 [ 6770.553730] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6770.568222] LustreError: 199665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6770.961355] Lustre: server umount lustre-MDT0001 complete [ 6794.902827] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 15:52:10 (1789069930) [ 6814.692584] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 6826.033855] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 6861.092930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6861.398638] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6861.458757] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6861.527425] Lustre: lustre-MDT0000: new disk, initializing [ 6861.626313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6861.669375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6866.470801] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6876.987576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6877.150862] Lustre: 207764:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6877.213430] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6877.219380] Lustre: Skipped 1 previous similar message [ 6877.313757] Lustre: lustre-MDT0001: new disk, initializing [ 6877.430630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6877.437600] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6881.968645] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6891.717529] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6898.648294] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6905.396740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6905.737767] Lustre: lustre-OST0000: new disk, initializing [ 6905.741913] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6905.749236] Lustre: 209735:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6911.437529] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6913.098656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6913.107880] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6913.212917] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6922.820860] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6929.185240] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6929.518430] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6934.757621] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6935.028997] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6939.108397] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6939.117180] LustreError: Skipped 2 previous similar messages [ 6939.119926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6939.127758] Lustre: Skipped 1 previous similar message [ 6939.133355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6939.135850] Lustre: Skipped 10 previous similar messages [ 6943.580688] Lustre: server umount lustre-OST0000 complete [ 6953.938205] Lustre: server umount lustre-MDT0000 complete [ 6954.472585] LustreError: 207777:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6954.531201] LustreError: 207777:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 6958.664658] LustreError: 207759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789070095 with bad export cookie 10363759440829356617 [ 6958.666353] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6958.691050] LustreError: 207759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6959.060032] Lustre: server umount lustre-MDT0001 complete [ 6967.603934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6973.798641] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6982.890802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6987.649921] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6996.009130] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7002.405132] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7009.844763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7018.001910] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7025.818732] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7030.618919] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7030.854241] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7035.404552] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7035.658317] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7038.954156] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7038.971326] LustreError: Skipped 3 previous similar messages [ 7044.383928] Lustre: server umount lustre-OST0000 complete [ 7050.896222] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7056.298640] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7086.084937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7093.882751] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7094.726884] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7094.756653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7094.761287] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7101.587374] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7104.022374] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7110.113608] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7110.118370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7110.128574] Lustre: Skipped 6 previous similar messages [ 7110.138507] Lustre: Skipped 6 previous similar messages [ 7115.032099] Lustre: server umount lustre-OST0000 complete [ 7125.342426] Lustre: server umount lustre-MDT0000 complete [ 7125.475972] LustreError: 211956:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7125.500035] LustreError: 211956:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 7129.379628] LustreError: 211936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789070266 with bad export cookie 10363759440829357499 [ 7129.403666] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7129.897294] Lustre: server umount lustre-MDT0001 complete [ 7138.985624] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 15:57:54 (1789070274) [ 7145.393225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7145.693596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7145.699331] Lustre: Skipped 6 previous similar messages [ 7150.156144] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7159.188064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7165.680557] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7174.675624] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7180.937524] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7188.892239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7197.132492] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7205.679036] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7210.758410] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7211.033424] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7215.956305] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7216.334960] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7223.882623] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7224.228714] Lustre: lustre-OST0001: new disk, initializing [ 7224.235561] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7224.255738] Lustre: 221292:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7231.587733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7231.612507] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7231.683615] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7231.753821] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7242.148620] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7247.162583] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7247.385975] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7252.548609] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7252.780129] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7257.568229] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7257.580387] LustreError: Skipped 3 previous similar messages [ 7268.831135] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7268.942413] Lustre: server umount lustre-OST0001 complete [ 7274.970516] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7275.232208] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7280.257826] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7280.660036] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7312.266913] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7317.994559] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7318.013622] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7318.031993] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7318.057517] Lustre: Skipped 2 previous similar messages [ 7319.673306] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7327.920371] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7329.757802] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7346.149250] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7346.563679] Lustre: server umount lustre-OST0001 complete [ 7369.202404] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7369.219412] Lustre: Skipped 6 previous similar messages [ 7369.229668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7369.237060] Lustre: Skipped 18 previous similar messages [ 7373.194490] Lustre: server umount lustre-OST0000 complete [ 7384.540834] Lustre: server umount lustre-MDT0000 complete [ 7384.544916] LustreError: 222897:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7384.575337] LustreError: 222897:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 38 previous similar messages [ 7388.668462] LustreError: 217902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789070525 with bad export cookie 10363759440829358465 [ 7388.674885] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7388.679679] LustreError: 217902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7395.268568] Lustre: server umount lustre-MDT0001 complete [ 7419.281633] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 16:02:34 (1789070554) [ 7425.790605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7426.178927] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7426.231633] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7431.936396] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7441.548621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7441.613242] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7441.633459] Lustre: 226556:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7446.768568] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7455.336948] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7461.464888] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7468.175456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7468.395613] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7475.464833] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7482.967226] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7492.410740] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7492.642907] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7501.120200] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7509.864375] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7511.890985] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7517.653634] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7517.822064] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7524.642575] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7524.880391] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7532.538605] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8cd845863000.ost_server_uuid 50 [ 7535.528179] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8cd845863000.ost_server_uuid in IDLE state after 0 sec [ 7537.073866] Lustre: Permanently deactivating lustre-OST0001 [ 7537.132528] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7543.402740] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7546.733890] Lustre: Permanently reactivating lustre-OST0001 [ 7546.748853] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7546.754110] Lustre: Skipped 2 previous similar messages [ 7549.464959] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7549.481809] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7549.504730] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7549.597463] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7554.536492] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7554.544882] LustreError: Skipped 2 previous similar messages [ 7557.571956] Lustre: server umount lustre-OST0001 complete [ 7576.034214] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7576.434495] Lustre: server umount lustre-OST0000 complete [ 7587.098760] Lustre: server umount lustre-MDT0000 complete [ 7591.165886] LustreError: 226546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789070728 with bad export cookie 10363759440829359753 [ 7591.176878] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7591.190604] LustreError: 226546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7592.003480] Lustre: server umount lustre-MDT0001 complete [ 7623.242594] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 16:05:58 (1789070758) [ 7633.096407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7633.546514] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7633.603760] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7633.613485] Lustre: Skipped 2 previous similar messages [ 7640.216974] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7650.273865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7650.351134] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7650.373363] Lustre: 233264:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7657.458933] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7666.260579] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7672.263255] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7680.502393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7680.711247] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7680.742525] Lustre: Permanently deactivating lustre-OST0000 [ 7680.756970] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7680.928765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7680.937294] Lustre: Skipped 10 previous similar messages [ 7688.937855] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7690.286679] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7690.297025] Lustre: Skipped 1 previous similar message [ 7696.439838] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7703.498526] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7703.580627] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7711.632251] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7712.235211] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7721.431807] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7724.104204] Lustre: Permanently reactivating lustre-OST0000 [ 7724.139550] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7724.146303] Lustre: Skipped 2 previous similar messages [ 7732.363399] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7732.366067] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0001-mdtlov_UUID (at 0@lo) in 42 seconds. I think it's dead, and I am evicting it. exp 00000000e94aaa1d, cur 1789070869 deadline 1789070857 last 1789070827 [ 7732.476928] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7742.286973] Lustre: server umount lustre-OST0001 complete [ 7760.355383] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7760.552615] Lustre: server umount lustre-OST0000 complete [ 7780.319173] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7780.532640] Lustre: server umount lustre-MDT0000 complete [ 7784.778967] LustreError: 233256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789070921 with bad export cookie 10363759440829361559 [ 7784.792831] LustreError: 233256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7784.795086] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7785.695676] Lustre: server umount lustre-MDT0001 complete [ 7797.890247] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 7801.075460] Key type lgssc unregistered [ 7801.398597] LNet: 238125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7801.402924] LNetError: 238125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7801.415904] LNet: Removed LNI 192.168.202.102@tcp [ 7802.461267] Key type .llcrypt unregistered [ 7802.464597] Key type ._llcrypt unregistered [ 7829.045573] Key type ._llcrypt registered [ 7829.050031] Key type .llcrypt registered [ 7829.187717] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 7830.391946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7830.440892] alg: No test for adler32 (adler32-zlib) [ 7831.909623] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 7832.346519] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 7834.192228] Key type lgssc registered [ 7835.575463] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7845.479977] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7845.500828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7846.949202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7852.275821] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7856.106133] LustreError: 239586:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7856.131585] LustreError: 239586:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7861.221049] LustreError: 239587:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7861.483085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7861.624996] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7861.668644] Lustre: 239574:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7861.679906] Lustre: Permanently deactivating lustre-MDT0001 [ 7861.697892] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7862.069875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7867.539409] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7868.994181] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7876.532282] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7882.553545] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7894.537048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7895.476537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7903.963419] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7906.752599] LustreError: 241427:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7911.911357] LustreError: 241429:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7913.344045] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7916.056544] LustreError: 241430:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7916.073603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7916.079550] LustreError: 241430:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7922.167255] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7922.522704] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7927.812310] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7927.833555] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7929.712198] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7938.244496] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7941.681806] Lustre: Permanently reactivating lustre-MDT0001 [ 7941.694961] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7941.697860] Lustre: Skipped 2 previous similar messages [ 7947.465692] Lustre: lustre-MDT0001: Client fc8643ef-bfa8-4ad7-b0d6-ede77f356f89 (at 192.168.202.2@tcp) reconnecting [ 7947.937401] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7947.957679] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7947.981819] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7948.015183] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7948.016265] LustreError: 243156:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7948.042409] LustreError: 243156:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7948.054387] LustreError: 243156:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.102@tcp: cfg command failed: rc = -108 [ 7948.067184] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7948.067184] [ 7948.080624] LustreError: 239579:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7954.155550] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7955.782255] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7962.790986] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7963.078285] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7968.773297] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7969.069453] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7972.466196] Lustre: Permanently deactivating lustre-MDT0001 [ 7972.495092] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7972.504845] Lustre: Skipped 2 previous similar messages [ 7979.730508] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7986.689111] Lustre: server umount lustre-MDT0000 complete [ 7989.222091] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7989.222921] LustreError: 239581:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7989.247524] Lustre: Skipped 2 previous similar messages [ 7989.279657] LustreError: 239581:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 7991.519693] LustreError: 241448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789071128 with bad export cookie 14913371565551295507 [ 7991.525519] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7991.531387] LustreError: 241448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7991.970120] Lustre: server umount lustre-MDT0001 complete [ 7996.945855] Lustre: server umount lustre-OST0000 complete [ 8011.695172] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 16:12:26 (1789071146) [ 8024.589258] Lustre: server umount lustre-OST0001 complete [ 8032.554496] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 8043.532227] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 8087.592783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8087.837677] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8087.853041] Lustre: Skipped 2 previous similar messages [ 8087.885831] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8087.952303] Lustre: lustre-MDT0000: new disk, initializing [ 8088.008441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8088.019898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8094.485162] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8109.119559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8109.275300] Lustre: 247852:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8109.357898] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8109.361775] Lustre: Skipped 1 previous similar message [ 8109.439633] Lustre: lustre-MDT0001: new disk, initializing [ 8109.595075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8109.656230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8109.671124] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8115.228645] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8126.029178] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8132.884352] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8139.784783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8140.321334] Lustre: lustre-OST0000: new disk, initializing [ 8140.324738] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8140.330766] Lustre: 249824:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8140.430644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8146.998115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8147.005279] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8147.140395] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8148.706986] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8160.180360] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8162.326825] LustreError: 247860:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8172.405769] LustreError: 247860:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8172.437370] LustreError: 247858:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8172.878694] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8173.224287] Lustre: lustre-OST0001: new disk, initializing [ 8173.227118] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8173.235665] Lustre: 250940:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8173.332723] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8181.584587] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8181.931447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8181.942480] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8182.073588] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8182.543113] LustreError: 247858:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8193.043335] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8195.037258] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8197.601792] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8197.602445] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8197.617964] LustreError: Skipped 1 previous similar message [ 8197.631682] Lustre: Skipped 1 previous similar message [ 8197.635695] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8197.650326] Lustre: Skipped 1 previous similar message [ 8202.726636] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8202.740537] Lustre: Skipped 1 previous similar message [ 8203.248797] Lustre: server umount lustre-OST0001 complete [ 8207.846098] LustreError: 249805:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8207.855203] LustreError: 249805:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8223.209290] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8223.220417] Lustre: Skipped 2 previous similar messages [ 8223.220696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8228.324066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8228.336565] Lustre: Skipped 1 previous similar message [ 8229.627659] Lustre: server umount lustre-OST0000 complete [ 8237.537005] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8237.543088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8237.552362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8237.556395] Lustre: Skipped 1 previous similar message [ 8248.804103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8248.808432] Lustre: Skipped 5 previous similar messages [ 8249.312108] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8249.480597] Lustre: server umount lustre-MDT0000 complete [ 8253.923413] LustreError: 247864:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8253.949427] LustreError: 247864:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 8254.105101] LustreError: 249823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789071391 with bad export cookie 14913371565551298097 [ 8254.107293] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8254.111419] LustreError: 249823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8261.411325] Lustre: server umount lustre-MDT0001 complete [ 8275.210846] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 8278.710831] Key type lgssc unregistered [ 8279.296679] LNet: 252977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8279.315506] LNetError: 252977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8279.343766] LNet: Removed LNI 192.168.202.102@tcp [ 8280.486649] Key type .llcrypt unregistered [ 8280.493186] Key type ._llcrypt unregistered [ 8330.223555] Key type ._llcrypt registered [ 8330.227358] Key type .llcrypt registered [ 8330.384870] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 8331.756546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8331.998735] alg: No test for adler32 (adler32-zlib) [ 8333.183601] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 8333.489133] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 8335.320299] Key type lgssc registered [ 8336.894664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8347.465778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8347.489934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8349.110918] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8349.221538] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8349.647719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8357.118978] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8367.714188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8367.866362] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8367.960343] Lustre: 255184:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8368.409887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8374.360826] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8381.451711] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8387.652807] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8395.051356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8395.344901] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8395.633425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8402.618841] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8404.156048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8412.778071] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8471.030259] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8471.047149] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8471.053159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8471.080523] Lustre: Skipped 1 previous similar message [ 8476.151369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8476.160536] Lustre: Skipped 2 previous similar messages [ 8477.136900] Lustre: server umount lustre-OST0000 complete [ 8482.637860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8498.500553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8499.083706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8501.113964] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8501.149152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8501.151635] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8501.192112] Lustre: Skipped 1 previous similar message [ 8507.059885] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8515.504044] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8557.545088] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8557.553970] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8557.559544] LustreError: Skipped 1 previous similar message [ 8557.570041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8557.589261] Lustre: Skipped 1 previous similar message [ 8562.667570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8562.673367] Lustre: Skipped 1 previous similar message [ 8562.790125] Lustre: server umount lustre-OST0000 complete [ 8567.779959] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8567.784268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8567.807962] Lustre: Skipped 2 previous similar messages [ 8574.002404] Lustre: server umount lustre-MDT0000 complete [ 8580.838248] LustreError: 255175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789071717 with bad export cookie 17123820109164534201 [ 8580.856390] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8580.866823] LustreError: 255175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8582.046806] Lustre: server umount lustre-MDT0001 complete [ 8595.707292] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 8600.130171] Key type lgssc unregistered [ 8600.384748] LNet: 260610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8600.388491] LNetError: 260610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8601.448121] LNet: Removed LNI 192.168.202.102@tcp [ 8603.336601] Key type .llcrypt unregistered [ 8603.342247] Key type ._llcrypt unregistered [ 8634.328273] Key type ._llcrypt registered [ 8634.329979] Key type .llcrypt registered [ 8634.438732] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 8635.718262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8635.765283] alg: No test for adler32 (adler32-zlib) [ 8636.922584] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 8637.144770] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 8638.783255] Key type lgssc registered [ 8639.783381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8646.755096] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8646.772283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8648.316740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8653.281947] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8658.937506] LustreError: 262026:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8658.977406] LustreError: 262026:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8664.053413] LustreError: 262025:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8665.556309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8666.496604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8672.345639] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8684.916862] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8692.516768] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8700.674698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8701.000960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8707.874531] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8717.362841] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8718.448578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8747.491144] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8747.506388] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8747.523596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8749.566400] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8749.576321] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8749.577925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8753.312441] Lustre: server umount lustre-OST0000 complete [ 8758.754819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8758.761798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8758.774059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8758.778461] Lustre: Skipped 1 previous similar message [ 8764.246534] Lustre: server umount lustre-MDT0000 complete [ 8764.913357] LustreError: 264144:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8764.943681] LustreError: 264144:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8768.602410] LustreError: 262007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789071905 with bad export cookie 10031666091509416507 [ 8768.610199] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8768.618292] LustreError: 262007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8769.385707] Lustre: server umount lustre-MDT0001 complete [ 8783.962720] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 8787.291761] Key type lgssc unregistered [ 8787.651253] LNet: 266332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8787.656361] LNetError: 266332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8788.712801] LNet: Removed LNI 192.168.202.102@tcp [ 8789.850958] Key type .llcrypt unregistered [ 8789.854814] Key type ._llcrypt unregistered [ 8811.129361] Key type ._llcrypt registered [ 8811.133702] Key type .llcrypt registered [ 8811.252863] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 8812.510345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8812.622161] alg: No test for adler32 (adler32-zlib) [ 8814.054521] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 8814.480256] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 8816.183298] Key type lgssc registered [ 8818.498047] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8828.496086] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8828.511799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8829.757255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8833.508517] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8839.149769] LustreError: 267566:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8839.175913] LustreError: 267566:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8843.197377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8843.886794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8849.994564] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8859.415704] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8866.178301] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8874.299646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8874.666130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8875.745655] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8881.922531] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8890.310727] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8901.602537] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8901.606800] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8901.616020] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8903.650716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8903.651463] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8907.851205] Lustre: server umount lustre-OST0000 complete [ 8913.891698] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8913.909137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8913.919587] Lustre: Skipped 2 previous similar messages [ 8918.860100] Lustre: server umount lustre-MDT0000 complete [ 8919.032913] LustreError: 268697:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8919.060888] LustreError: 268697:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8922.421827] LustreError: 267547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789072059 with bad export cookie 15502996275809022497 [ 8922.427863] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8922.850807] Lustre: server umount lustre-MDT0001 complete [ 8936.638607] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 8940.600792] Key type lgssc unregistered [ 8941.111952] LNet: 271246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8941.120442] LNetError: 271246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8941.150386] LNet: Removed LNI 192.168.202.102@tcp [ 8942.396318] Key type .llcrypt unregistered [ 8942.397576] Key type ._llcrypt unregistered [ 8975.347439] Key type ._llcrypt registered [ 8975.348823] Key type .llcrypt registered [ 8975.571038] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 8977.349018] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8977.539429] alg: No test for adler32 (adler32-zlib) [ 8978.796881] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 8979.085727] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 8980.855396] Key type lgssc registered [ 8982.131809] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8992.379611] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8992.396992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8993.970869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8999.305512] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9003.493402] LustreError: 272663:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9003.521528] LustreError: 272663:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9008.631299] LustreError: 272662:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9012.777554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9013.522421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9019.876320] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9029.528231] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9037.683496] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9048.067112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9048.730636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9055.459082] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9063.151507] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 9063.443878] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9094.119046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9094.122907] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9094.128069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9094.141187] Lustre: Skipped 1 previous similar message [ 9098.015897] Lustre: server umount lustre-OST0000 complete [ 9104.354808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9104.360593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9104.377728] Lustre: Skipped 1 previous similar message [ 9104.378785] Lustre: Skipped 1 previous similar message [ 9108.062498] Lustre: server umount lustre-MDT0000 complete [ 9109.477886] LustreError: 272658:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9109.499921] LustreError: 272658:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9112.490796] LustreError: 272644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789072249 with bad export cookie 10324794382400239545 [ 9112.493126] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9112.509324] LustreError: 272644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9113.086722] Lustre: server umount lustre-MDT0001 complete [ 9126.881765] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 9131.516634] Key type lgssc unregistered [ 9131.992761] LNet: 276970:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9132.016688] LNetError: 276970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9132.069332] LNet: Removed LNI 192.168.202.102@tcp [ 9134.253291] Key type .llcrypt unregistered [ 9134.256194] Key type ._llcrypt unregistered [ 9158.482459] Key type ._llcrypt registered [ 9158.487408] Key type .llcrypt registered [ 9158.698765] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 9160.103426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9160.207117] alg: No test for adler32 (adler32-zlib) [ 9161.361994] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 9161.819504] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 9163.610358] Key type lgssc registered [ 9165.293444] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9176.187119] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9176.209342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9177.885420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9183.776373] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9187.302506] LustreError: 278204:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9187.335370] LustreError: 278204:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9192.417927] LustreError: 278206:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9192.450142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9192.762207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9197.688043] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9207.852165] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9215.765115] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9224.122644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9224.972513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9231.303272] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9238.719297] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9247.661361] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9268.192308] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9268.195697] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9268.202482] LustreError: Skipped 1 previous similar message [ 9268.203624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9268.221573] Lustre: Skipped 1 previous similar message [ 9268.236906] Lustre: Skipped 1 previous similar message [ 9272.293115] Lustre: server umount lustre-OST0000 complete [ 9278.432695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9278.451602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9282.822095] Lustre: server umount lustre-MDT0000 complete [ 9283.554194] LustreError: 278194:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9283.590795] LustreError: 278194:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9286.340827] LustreError: 278170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789072423 with bad export cookie 5057263654450774242 [ 9286.345602] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9286.350721] LustreError: 278170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9286.710685] Lustre: server umount lustre-MDT0001 complete [ 9297.854780] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 9301.405836] Key type lgssc unregistered [ 9301.739784] LNet: 281930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9301.744155] LNetError: 281930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9301.759938] LNet: Removed LNI 192.168.202.102@tcp [ 9302.747244] Key type .llcrypt unregistered [ 9302.749535] Key type ._llcrypt unregistered [ 9314.372889] Key type ._llcrypt registered [ 9314.384859] Key type .llcrypt registered [ 9314.537556] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9337.545264] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 9358.126550] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 9359.491313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9359.577153] alg: No test for adler32 (adler32-zlib) [ 9360.806905] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 9361.096489] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 9362.800621] Key type lgssc registered [ 9364.638529] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9409.655238] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9409.728223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9411.984678] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9412.044218] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9412.232090] Lustre: lustre-MDT0000: new disk, initializing [ 9412.371872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9412.399539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9418.470460] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9431.189148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9431.302242] Lustre: 285483:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9431.457328] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9431.461016] Lustre: Skipped 1 previous similar message [ 9431.524728] Lustre: lustre-MDT0001: new disk, initializing [ 9431.625880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9431.654421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9431.662940] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9437.638364] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9449.960525] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9456.724559] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9464.133144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9464.644769] Lustre: lustre-OST0000: new disk, initializing [ 9464.662810] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9464.690915] Lustre: 287452:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9464.914846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9471.081486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9471.095952] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9471.227269] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9471.285125] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9481.961865] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9489.770969] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9490.352254] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9495.993245] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9496.313280] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9502.189593] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9502.194730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9502.199653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9502.216483] Lustre: Skipped 1 previous similar message [ 9505.363933] Lustre: server umount lustre-OST0000 complete [ 9512.418730] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9512.428359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9512.440526] Lustre: Skipped 2 previous similar messages [ 9512.453469] Lustre: Skipped 1 previous similar message [ 9517.475709] Lustre: server umount lustre-MDT0000 complete [ 9522.664309] LustreError: 285494:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9522.695544] LustreError: 285494:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 9522.914793] LustreError: 285475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789072659 with bad export cookie 13240415865557838281 [ 9522.929910] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9522.943854] LustreError: 285475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9523.524454] Lustre: server umount lustre-MDT0001 complete [ 9532.864188] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 16:37:48 (1789072668) [ 9539.892239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9540.330806] LustreError: 289864:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9540.416926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9544.022737] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9545.715447] LustreError: 289865:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9550.825831] LustreError: 289864:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9554.186745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9554.510076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9562.157863] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9574.216632] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9579.709907] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9587.818470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9588.556959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9595.567536] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9602.894571] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9625.574811] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9625.595023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9625.617180] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9625.637873] LustreError: Skipped 1 previous similar message [ 9629.054799] Lustre: server umount lustre-OST0000 complete [ 9635.811073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9635.816974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9635.842298] Lustre: Skipped 2 previous similar messages [ 9635.871284] Lustre: Skipped 2 previous similar messages [ 9638.999891] Lustre: server umount lustre-MDT0000 complete [ 9640.931322] LustreError: 289864:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9644.885495] LustreError: 289845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789072781 with bad export cookie 13240415865557839163 [ 9644.915673] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9645.830508] Lustre: server umount lustre-MDT0001 complete [ 9661.646990] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 9665.117480] Key type lgssc unregistered [ 9665.451401] LNet: 293425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9665.456255] LNetError: 293425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9665.489761] LNet: Removed LNI 192.168.202.102@tcp [ 9666.729935] Key type .llcrypt unregistered [ 9666.735242] Key type ._llcrypt unregistered [ 9704.611438] Key type ._llcrypt registered [ 9704.619355] Key type .llcrypt registered [ 9704.914308] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 9726.250713] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 9727.927606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9728.038267] alg: No test for adler32 (adler32-zlib) [ 9729.347677] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [ 9729.631259] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 9731.386558] Key type lgssc registered [ 9733.018209] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9794.732556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9794.777237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9796.116382] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9796.167144] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9796.227479] Lustre: lustre-MDT0000: new disk, initializing [ 9796.304763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9796.338275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9802.151443] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9816.302896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9816.529640] Lustre: 297256:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9816.620960] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9816.630140] Lustre: Skipped 1 previous similar message [ 9816.759095] Lustre: lustre-MDT0001: new disk, initializing [ 9816.988852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9817.064702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9817.096804] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9822.940653] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9834.332713] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9840.667894] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9847.221262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9847.609611] Lustre: lustre-OST2710: new disk, initializing [ 9847.614653] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9847.626087] Lustre: 299223:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9847.827354] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9854.078841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9854.110476] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9854.394235] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9857.705545] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9869.928450] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9879.233435] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9879.547845] Lustre: lustre-OST03e8: new disk, initializing [ 9879.558575] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9879.577803] Lustre: 300289:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9879.865780] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9887.034479] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9887.837117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9887.842195] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9887.961956] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9899.190610] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9905.877222] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9906.054578] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9911.758067] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9912.289726] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9920.356986] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9920.669427] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9927.884308] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9928.237978] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9934.310536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9934.322104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9934.330718] Lustre: Skipped 3 previous similar messages [ 9934.345236] Lustre: Skipped 3 previous similar messages [ 9937.931187] Lustre: server umount lustre-MDT0000 complete [ 9942.765167] LustreError: 299222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789073079 with bad export cookie 5064507962571876774 [ 9942.767444] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9942.778051] LustreError: 299222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9943.061589] Lustre: server umount lustre-MDT0001 complete [ 9959.246495] Lustre: server umount lustre-OST2710 complete [ 9960.927789] Lustre: 295050:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789073081/real 1789073081] req@00000000f9280167 x1875978869615488/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789073097 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9960.955410] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9964.001112] Lustre: 295051:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789073084/real 1789073084] req@00000000647572e1 x1875978869615744/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789073100 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9965.084810] Lustre: server umount lustre-OST03e8 complete [ 9986.225178] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 9998.921406] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [10040.283991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10040.432147] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10040.462368] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10040.589992] Lustre: lustre-MDT0000: new disk, initializing [10040.764901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10040.806902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10045.259132] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10058.618047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10058.805908] Lustre: 305546:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10058.877050] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10058.882104] Lustre: Skipped 1 previous similar message [10058.981817] Lustre: lustre-MDT0001: new disk, initializing [10059.166638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10059.258375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10059.272663] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10064.882966] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10078.034675] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10084.459819] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10091.592323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10091.906358] Lustre: lustre-OST0000: new disk, initializing [10091.913656] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10091.918791] Lustre: 307514:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10092.045277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10097.240935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10097.249516] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10097.344101] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10098.851995] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10108.425119] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10113.462476] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10113.850319] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10119.218501] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10119.338593] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10123.234990] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10123.240253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10123.243747] Lustre: Skipped 1 previous similar message [10127.997423] Lustre: server umount lustre-OST0000 complete [10133.473025] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10133.490385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10133.502778] Lustre: Skipped 2 previous similar messages [10138.593746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10138.600788] Lustre: Skipped 1 previous similar message [10139.061179] Lustre: server umount lustre-MDT0000 complete [10143.137564] LustreError: 305538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789073280 with bad export cookie 5064507962571878454 [10143.141179] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10143.153710] LustreError: 305538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10143.689846] Lustre: server umount lustre-MDT0001 complete [10155.356535] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 16:48:09 (1789073289) [10157.477495] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [10159.760991] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 16:48:14 (1789073294) [10164.404637] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [10197.865413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10198.285614] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10198.310739] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10198.566595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10203.988224] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10214.181320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10214.247991] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10214.275207] Lustre: 311370:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10214.349849] LustreError: 311381:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10214.354944] LustreError: 311381:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10219.901409] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10245.797808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10245.951138] Lustre: Denying initial registration attempt from nid 192.168.202.102@tcp, specified as failover [10245.968749] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10245.977745] LustreError: 312825:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [10245.980893] LustreError: 312825:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [10245.983713] LustreError: 312825:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10246.029236] Lustre: server umount lustre-OST0000 complete [10246.033849] LustreError: 312825:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [10255.328837] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10255.332966] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10255.339746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10255.349028] Lustre: Skipped 2 previous similar messages [10265.569617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10265.573780] Lustre: Skipped 4 previous similar messages [10268.127136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10268.477799] Lustre: server umount lustre-MDT0000 complete [10270.693792] LustreError: 311377:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10270.732452] LustreError: 311377:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10273.803491] LustreError: 311361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789073410 with bad export cookie 5064507962571879336 [10273.807663] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10273.814586] LustreError: 311361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10274.458341] Lustre: server umount lustre-MDT0001 complete [10293.830159] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [10297.767994] Key type lgssc unregistered [10298.319905] LNet: 313963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10298.325460] LNetError: 313963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10299.393907] LNet: Removed LNI 192.168.202.102@tcp [10300.940598] Key type .llcrypt unregistered [10300.941808] Key type ._llcrypt unregistered [10317.384501] Key type ._llcrypt registered [10317.385846] Key type .llcrypt registered [10317.439364] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [10318.610646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10318.655657] alg: No test for adler32 (adler32-zlib) [10319.823934] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [10320.001175] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [10321.655265] Key type lgssc registered [10323.110805] Lustre: Echo OBD driver; http://www.lustre.org/ [10375.186535] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [10382.918461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10382.941662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10384.306120] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10384.324522] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10384.496350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10389.529256] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10402.677057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10402.846159] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10402.881127] Lustre: 316724:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10403.258841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10409.752656] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10422.409374] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10429.020514] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10451.627467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10451.844481] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10452.049493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10461.133933] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10472.359731] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10479.087732] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10479.097444] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10479.142291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10479.154419] Lustre: Skipped 1 previous similar message [10480.399395] Lustre: server umount lustre-OST0000 complete [10484.196171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10484.197689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10484.207246] Lustre: Skipped 2 previous similar messages [10484.213138] Lustre: Skipped 1 previous similar message [10489.313350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10489.327422] Lustre: Skipped 1 previous similar message [10490.582721] Lustre: server umount lustre-MDT0000 complete [10493.977397] LustreError: 316719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789073630 with bad export cookie 10397715742624878377 [10493.989129] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10493.998961] LustreError: 316719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10494.281536] Lustre: server umount lustre-MDT0001 complete [10505.968552] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [10508.770989] Key type lgssc unregistered [10509.091591] LNet: 320304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10509.097907] LNetError: 320304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10509.107167] LNet: Removed LNI 192.168.202.102@tcp [10510.151449] Key type .llcrypt unregistered [10510.153094] Key type ._llcrypt unregistered [10534.267271] Key type ._llcrypt registered [10534.268770] Key type .llcrypt registered [10534.356690] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [10535.841113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10535.898733] alg: No test for adler32 (adler32-zlib) [10537.354600] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [10537.656202] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [10539.391577] Key type lgssc registered [10541.201662] Lustre: Echo OBD driver; http://www.lustre.org/ [10551.559597] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10551.595341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10553.337888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10558.668679] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10563.557231] LustreError: 321719:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10563.571141] LustreError: 321719:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10567.090082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10567.573834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10571.301403] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10579.773217] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10587.470364] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10596.725344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10597.207659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10603.804120] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10612.268774] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10622.951092] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10622.954056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10622.966963] Lustre: Skipped 2 previous similar messages [10622.971438] Lustre: Skipped 1 previous similar message [10625.215091] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.2@tcp (stopping) [10627.446461] Lustre: server umount lustre-MDT0000 complete [10628.065944] LustreError: 321718:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10628.083941] LustreError: 321718:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10630.285454] LustreError: 321713:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10631.676839] LustreError: 321698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789073768 with bad export cookie 7179372226016103439 [10631.685994] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10631.698321] LustreError: 321698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10633.184696] LustreError: 321714:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10633.186082] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10633.233056] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10635.442135] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.2@tcp (stopping) [10638.104994] Lustre: server umount lustre-MDT0001 complete [10645.266339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10653.761971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10656.738053] LustreError: 321025:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000aa6614be x1875979716830720/t0(0) o250->MGC192.168.202.102@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10657.179990] LustreError: 323550:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10657.201476] LustreError: 323550:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10657.301774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10657.372911] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10661.870980] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10666.467776] LustreError: 323551:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10666.484486] LustreError: 323551:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10667.516715] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10671.571588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10671.978953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10677.104850] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10677.233149] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10683.489655] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10685.276349] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10691.014940] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10692.600917] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10698.210425] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10698.223082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10698.239305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10698.247789] Lustre: Skipped 1 previous similar message [10702.821024] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10702.862294] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10704.362224] Lustre: server umount lustre-OST0000 complete [10713.060331] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10713.062747] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10713.072796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10713.072815] Lustre: Skipped 2 previous similar messages [10713.086694] Lustre: Skipped 1 previous similar message [10716.090547] Lustre: server umount lustre-MDT0000 complete [10718.192403] LustreError: 327237:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10718.222890] LustreError: 327237:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10721.089891] LustreError: 323563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789073858 with bad export cookie 7179372226016111153 [10721.100308] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10721.103031] LustreError: 323563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10721.512928] Lustre: server umount lustre-MDT0001 complete [10736.279544] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [10738.369892] Key type lgssc unregistered [10738.820630] LNet: 327987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10738.849220] LNetError: 327987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10739.881266] LNet: Removed LNI 192.168.202.102@tcp [10741.412140] Key type .llcrypt unregistered [10741.419683] Key type ._llcrypt unregistered [10771.227177] Key type ._llcrypt registered [10771.229386] Key type .llcrypt registered [10771.422615] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [10772.851297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10772.877809] alg: No test for adler32 (adler32-zlib) [10774.059217] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [10774.393810] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [10776.111272] Key type lgssc registered [10777.975992] Lustre: Echo OBD driver; http://www.lustre.org/ [10788.245375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10788.259252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10790.289326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10795.927327] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10800.610666] LustreError: 329402:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10800.664628] LustreError: 329402:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10805.731773] LustreError: 329403:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10808.036546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10808.570851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10814.537661] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10823.356495] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10829.428034] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10833.381410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10833.386376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10833.392985] Lustre: Skipped 1 previous similar message [10833.404373] Lustre: Skipped 1 previous similar message [10834.408315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10834.414708] Lustre: Skipped 1 previous similar message [10838.981962] Lustre: server umount lustre-MDT0000 complete [10839.522179] LustreError: 329399:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10839.547929] LustreError: 329399:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10842.792841] LustreError: 329381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789073979 with bad export cookie 15285297441652731033 [10842.803530] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10842.815396] LustreError: 329381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10843.279451] Lustre: server umount lustre-MDT0001 complete [10865.295364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10865.579428] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10865.598236] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10865.763553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10872.214947] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10884.158717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10884.492934] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10884.571532] Lustre: 331989:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10885.128311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10891.299404] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10902.769535] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10910.905771] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10923.061867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10923.512200] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10923.524823] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10923.532635] LustreError: 333878:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [10923.538988] LustreError: 333878:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [10923.569766] LustreError: 333878:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10923.630503] Lustre: server umount lustre-OST0000 complete [10923.631948] LustreError: 333878:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10934.068239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10934.387564] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10934.677388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10939.819206] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10941.676950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10948.124584] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10956.499668] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10956.627049] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10956.715337] Lustre: lustre-OST0001: new disk, initializing [10956.726282] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10956.739846] Lustre: 335429:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10956.918643] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10963.078350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10963.081783] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10963.103188] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10963.211197] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10970.644181] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10978.800654] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10978.803544] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10978.806039] LustreError: Skipped 1 previous similar message [10978.807759] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10978.829534] Lustre: Skipped 1 previous similar message [10980.215397] Lustre: server umount lustre-OST0001 complete [10983.965596] LustreError: 334357:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10983.995811] LustreError: 334357:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10988.000683] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10988.003208] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10988.004679] LustreError: Skipped 1 previous similar message [10988.007568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10988.021420] Lustre: Skipped 1 previous similar message [10988.024281] Lustre: Skipped 1 previous similar message [10989.036464] LustreError: 334357:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10989.053813] LustreError: 334357:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10990.949732] Lustre: server umount lustre-OST0000 complete [10997.728753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10997.733651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10997.751112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10997.758476] Lustre: Skipped 3 previous similar messages [11001.006800] Lustre: server umount lustre-MDT0000 complete [11004.229194] LustreError: 331983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789074141 with bad export cookie 15285297441652731593 [11004.232121] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11004.236633] LustreError: 331983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11004.617360] Lustre: server umount lustre-MDT0001 complete [11015.551816] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [11017.947112] Key type lgssc unregistered [11018.252683] LNet: 337386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11018.261561] LNetError: 337386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11018.269819] LNet: Removed LNI 192.168.202.102@tcp [11018.879531] Key type .llcrypt unregistered [11018.885153] Key type ._llcrypt unregistered [11091.145927] Key type ._llcrypt registered [11091.147436] Key type .llcrypt registered [11091.260852] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [11107.462460] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [11108.863728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11108.919059] alg: No test for adler32 (adler32-zlib) [11110.207515] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [11110.586432] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [11112.215650] Key type lgssc registered [11113.552991] Lustre: Echo OBD driver; http://www.lustre.org/ [11156.461321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11156.497780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11157.889211] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11157.975640] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11158.136874] Lustre: lustre-MDT0000: new disk, initializing [11158.263965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11158.299149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11164.126511] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11174.047617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11174.151833] Lustre: 342320:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11174.200796] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11174.204542] Lustre: Skipped 1 previous similar message [11174.262885] Lustre: lustre-MDT0001: new disk, initializing [11174.356494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11174.390215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11174.404551] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11178.773738] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11188.280950] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11195.028344] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11202.839431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11203.116388] Lustre: lustre-OST0000: new disk, initializing [11203.121219] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11203.126533] Lustre: 344288:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11203.211624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11210.363914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11210.378619] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11210.665863] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11211.654396] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11226.226846] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11232.351115] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11232.580780] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11237.278511] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11237.492903] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11241.443342] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11241.452359] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11241.478612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11241.488065] Lustre: Skipped 1 previous similar message [11245.738269] Lustre: server umount lustre-OST0000 complete [11251.168980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11251.176581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11251.188135] Lustre: Skipped 1 previous similar message [11251.203530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11255.898138] Lustre: server umount lustre-MDT0000 complete [11259.882765] LustreError: 342312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789074396 with bad export cookie 2340135741261382059 [11259.891106] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11259.895654] LustreError: 342312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11260.353272] Lustre: server umount lustre-MDT0001 complete [11267.777451] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 17:06:43 (1789074403) [11275.230778] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 17:06:50 (1789074410) [11285.052335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11285.439945] LustreError: 347045:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11285.469123] LustreError: 347045:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11285.533703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11289.506760] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11290.595094] LustreError: 347046:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11295.719156] LustreError: 347045:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11298.241541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11298.715619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11303.996532] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11312.533170] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11318.670897] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11324.758087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11325.198435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11330.579461] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11338.151724] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11349.702444] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11351.417257] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11352.962235] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11354.381237] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11356.193277] Lustre: Failing over lustre-MDT0000 [11356.543527] Lustre: server umount lustre-MDT0000 complete [11360.228121] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11360.230738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11360.247568] Lustre: Skipped 1 previous similar message [11360.257961] LustreError: 347045:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11360.282746] LustreError: 347045:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11365.375284] LustreError: 347042:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11365.399664] LustreError: 347042:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11375.124664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11375.250299] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11375.565029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11375.632405] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11375.755780] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11380.293488] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11380.734548] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11380.783709] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [11380.853832] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11389.337429] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11390.764532] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11393.851526] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11395.375799] Lustre: Failing over lustre-MDT0000 [11395.564366] Lustre: server umount lustre-MDT0000 complete [11396.067017] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11396.072281] LustreError: 347041:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11396.115282] Lustre: Skipped 3 previous similar messages [11396.134986] LustreError: 347041:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [11412.320726] Lustre: 340542:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789074533/real 1789074533] req@00000000d004919b x1875980317713280/t0(0) o400->MGC192.168.202.102@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789074549 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11412.353571] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11412.378201] LustreError: 349488:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11412.412488] LustreError: 349488:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 16 previous similar messages [11416.023378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11422.688513] LustreError: 340540:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000096c658a7 x1875980317718528/t0(0) o250->MGC192.168.202.102@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11422.871470] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.2@tcp (not set up) [11422.883060] Lustre: Skipped 1 previous similar message [11423.230706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11423.342207] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11428.001186] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11428.350089] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11428.368511] Lustre: Skipped 2 previous similar messages [11428.420935] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [11428.501883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11429.835844] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11441.166104] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11442.783739] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11449.313860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11449.320768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11449.335284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11453.921873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11453.935100] Lustre: Skipped 2 previous similar messages [11455.768911] Lustre: server umount lustre-MDT0000 complete [11459.040971] LustreError: 347046:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11459.065319] LustreError: 347046:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [11460.114654] LustreError: 347027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789074597 with bad export cookie 2340135741261384768 [11460.126121] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11460.139424] LustreError: 347027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11460.680460] Lustre: server umount lustre-MDT0001 complete [11476.167609] Lustre: server umount lustre-OST0000 complete [11492.258386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11492.774817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11497.872320] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11506.472179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11511.586678] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11519.950519] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11525.914443] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11532.611473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11537.152740] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11539.525358] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11546.782628] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11556.833558] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11556.837931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11556.851337] Lustre: Skipped 3 previous similar messages [11556.869394] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11558.502740] Lustre: server umount lustre-OST0000 complete [11576.290681] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11576.297885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11576.314602] Lustre: Skipped 8 previous similar messages [11576.535676] Lustre: server umount lustre-MDT0000 complete [11580.898313] LustreError: 353378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789074717 with bad export cookie 2340135741261385517 [11580.906995] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11580.924254] LustreError: 353378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11581.331903] Lustre: server umount lustre-MDT0001 complete [11593.857988] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [11596.844843] Key type lgssc unregistered [11597.134302] LNet: 356927:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11597.138813] LNetError: 356927:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11598.189473] LNet: Removed LNI 192.168.202.102@tcp [11599.241722] Key type .llcrypt unregistered [11599.247906] Key type ._llcrypt unregistered [11628.926847] Key type ._llcrypt registered [11628.932459] Key type .llcrypt registered [11629.027828] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [11630.125808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11630.334681] alg: No test for adler32 (adler32-zlib) [11631.272908] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [11631.456181] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [11633.231536] Key type lgssc registered [11634.766928] Lustre: Echo OBD driver; http://www.lustre.org/ [11643.969518] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11643.999126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11645.596720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11650.755304] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11655.146995] LustreError: 358446:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11655.173295] LustreError: 358446:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11658.729961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11659.060736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11663.822527] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11672.093378] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11677.939584] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11685.277469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11685.567794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11690.690553] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11691.629767] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11699.279279] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11714.530336] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11714.534419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11714.569566] Lustre: Skipped 2 previous similar messages [11714.581179] Lustre: Skipped 2 previous similar messages [11715.718765] Lustre: server umount lustre-MDT0000 complete [11719.654777] LustreError: 358440:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11719.668806] LustreError: 358440:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11719.687286] LustreError: 358426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789074856 with bad export cookie 6382737970958988937 [11719.710132] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11719.987794] Lustre: server umount lustre-MDT0001 complete [11734.958832] Lustre: server umount lustre-OST0000 complete [11749.397272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11750.344214] LustreError: 362177:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11750.426753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11755.497896] LustreError: 362178:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11757.370155] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11760.609713] LustreError: 362177:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11767.575251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11768.370602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11772.521652] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11779.241229] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11785.367867] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11794.236147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11794.697150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11795.710416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11802.868612] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11813.079366] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11824.609548] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11824.620696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11824.643394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11828.712063] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11828.713683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11828.750749] Lustre: Skipped 1 previous similar message [11828.773869] Lustre: Skipped 2 previous similar messages [11831.004710] Lustre: server umount lustre-MDT0000 complete [11833.835171] LustreError: 362905:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11833.867035] LustreError: 362905:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11835.417383] LustreError: 362158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789074972 with bad export cookie 6382737970958990050 [11835.422981] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11835.445990] LustreError: 362158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [11835.929560] Lustre: server umount lustre-MDT0001 complete [11851.784450] Lustre: server umount lustre-OST0000 complete [11869.081020] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11869.485588] LustreError: 366088:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11869.555541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11873.674689] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11882.004213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11887.574717] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11897.161842] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11904.161198] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11911.437563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11911.915635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11911.922127] Lustre: Skipped 1 previous similar message [11915.007155] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11918.726830] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11925.920692] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11932.655616] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11932.659513] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11932.671025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11932.700384] Lustre: Skipped 1 previous similar message [11933.498143] Lustre: server umount lustre-OST0000 complete [11938.786301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11938.797171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11938.811450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11938.819057] Lustre: Skipped 1 previous similar message [11945.282432] Lustre: server umount lustre-MDT0000 complete [11948.000945] LustreError: 367930:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11948.047356] LustreError: 367930:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11950.553906] LustreError: 366070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789075087 with bad export cookie 6382737970958991072 [11950.559896] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11950.568176] LustreError: 366070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11951.081901] Lustre: server umount lustre-MDT0001 complete [11963.980729] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [11967.406374] Key type lgssc unregistered [11967.738736] LNet: 369622:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11967.748925] LNetError: 369622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11967.770615] LNet: Removed LNI 192.168.202.102@tcp [11969.034997] Key type .llcrypt unregistered [11969.039177] Key type ._llcrypt unregistered [12002.192581] Key type ._llcrypt registered [12002.194428] Key type .llcrypt registered [12002.426846] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [12004.592900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12005.039188] alg: No test for adler32 (adler32-zlib) [12006.542309] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [12006.930097] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [12008.655303] Key type lgssc registered [12010.348234] Lustre: Echo OBD driver; http://www.lustre.org/ [12019.459933] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12019.485022] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12019.490403] LustreError: 371061:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [12019.521314] LustreError: 371061:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [12019.536450] LustreError: 371061:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [12019.554557] LustreError: 371061:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [12019.562200] LustreError: 371061:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [12027.326775] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [12027.331664] LustreError: 371458:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [12027.346253] LustreError: 371458:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [12027.350672] LustreError: 371458:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [12027.355601] LustreError: 371458:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [12027.359602] LustreError: 371458:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [12053.958491] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [12056.207675] Key type lgssc unregistered [12056.477875] LNet: 372461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12056.483773] LNetError: 372461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12056.504821] LNet: Removed LNI 192.168.202.102@tcp [12057.533446] Key type .llcrypt unregistered [12057.535087] Key type ._llcrypt unregistered [12076.965256] Key type ._llcrypt registered [12076.972965] Key type .llcrypt registered [12077.106777] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [12095.408601] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [12096.864132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12097.001180] alg: No test for adler32 (adler32-zlib) [12098.153212] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [12098.537126] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [12100.255285] Key type lgssc registered [12101.499947] Lustre: Echo OBD driver; http://www.lustre.org/ [12139.556731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12139.592153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12141.011871] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12141.055462] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12141.226553] Lustre: lustre-MDT0000: new disk, initializing [12141.376105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12141.405824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12146.889432] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12159.871961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12160.045287] Lustre: 375620:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12160.139325] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12160.143529] Lustre: Skipped 1 previous similar message [12160.281355] Lustre: lustre-MDT0001: new disk, initializing [12160.365852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12160.401789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12160.419088] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12165.675965] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12175.990235] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12182.248065] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12190.921069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12191.342247] Lustre: lustre-OST0000: new disk, initializing [12191.347701] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12191.354172] Lustre: 377588:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12191.517355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12199.049123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12199.059478] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12199.228653] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12199.483759] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12209.727626] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12215.964982] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12216.293863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12221.946579] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12222.284934] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12224.999968] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12225.001934] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12225.038067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12225.070742] Lustre: Skipped 1 previous similar message [12230.131369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12230.138755] Lustre: Skipped 1 previous similar message [12235.234919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12235.254782] Lustre: Skipped 1 previous similar message [12239.328240] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12239.461636] Lustre: server umount lustre-OST0000 complete [12245.475883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12245.486633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12245.496940] Lustre: Skipped 2 previous similar messages [12250.264709] Lustre: server umount lustre-MDT0000 complete [12255.047554] LustreError: 375610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789075392 with bad export cookie 15196180972023759034 [12255.069126] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12255.387619] Lustre: server umount lustre-MDT0001 complete [12266.742847] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12269.094660] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 17:23:24 (1789075404) [12278.412896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12279.217295] LustreError: 380048:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12279.236284] LustreError: 380048:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12279.346788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12283.895490] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12284.394251] LustreError: 380049:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12289.518489] LustreError: 380048:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12294.626385] LustreError: 380049:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12296.374455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12297.312805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12304.135204] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12313.904565] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12321.452232] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12328.688370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12329.289501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12337.609824] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12348.352312] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12357.301210] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12357.452449] Lustre: lustre-OST0001: new disk, initializing [12357.455295] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12357.460344] Lustre: 382888:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12357.574755] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12363.548822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12363.584266] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12363.733669] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12370.155946] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12382.453143] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12389.347218] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12389.368260] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12389.397199] Lustre: Skipped 1 previous similar message [12389.410265] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12389.422126] Lustre: Skipped 1 previous similar message [12393.103320] Lustre: server umount lustre-OST0001 complete [12394.487321] LustreError: 381874:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12394.508982] LustreError: 381874:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12404.711516] LustreError: 383735:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12404.736816] LustreError: 383735:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [12414.946140] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12414.948761] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12414.961631] LustreError: Skipped 1 previous similar message [12414.965393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12414.982026] Lustre: Skipped 2 previous similar messages [12415.000894] Lustre: Skipped 2 previous similar messages [12420.434327] Lustre: server umount lustre-OST0000 complete [12430.311623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12430.313418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12430.350785] Lustre: Skipped 1 previous similar message [12431.864756] Lustre: server umount lustre-MDT0000 complete [12435.431971] LustreError: 380049:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12435.462183] LustreError: 380049:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [12436.699049] LustreError: 380030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789075573 with bad export cookie 15196180972023759916 [12436.701638] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12436.714057] LustreError: 380030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12451.299245] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12451.570272] Lustre: server umount lustre-MDT0001 complete [12465.787488] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [12469.457785] Key type lgssc unregistered [12469.899650] LNet: 384881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12469.911520] LNetError: 384881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12469.980495] LNet: Removed LNI 192.168.202.102@tcp [12471.800454] Key type .llcrypt unregistered [12471.804986] Key type ._llcrypt unregistered [12534.586271] Key type ._llcrypt registered [12534.588490] Key type .llcrypt registered [12534.721509] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [12536.225038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12536.402623] alg: No test for adler32 (adler32-zlib) [12537.818529] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [12537.980071] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [12539.658723] Key type lgssc registered [12540.983455] Lustre: Echo OBD driver; http://www.lustre.org/ [12550.276411] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12550.307438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12551.688284] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12551.728701] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12551.986515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12557.880057] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12567.737319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12567.966798] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12568.006633] Lustre: 387690:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12568.483714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12574.232834] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12583.700806] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12590.138896] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12594.160126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12594.172480] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12594.199137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12598.955358] Lustre: server umount lustre-MDT0000 complete [12603.163965] LustreError: 388434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789075740 with bad export cookie 15337048574270889418 [12603.164869] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12603.185752] LustreError: 388434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12603.730330] Lustre: server umount lustre-MDT0001 complete [12616.703393] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 17:29:11 (1789075751) [12624.910725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12625.520698] LustreError: 390207:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12625.555131] LustreError: 390207:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12625.645150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12631.010186] LustreError: 390208:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12631.090596] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12636.131593] LustreError: 390207:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12641.253729] LustreError: 390208:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12641.433000] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12641.967097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12647.883677] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12657.959733] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12665.520551] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12672.959187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12673.192707] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12673.508537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12681.136518] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12691.424883] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12701.216828] Lustre: Permanently deactivating lustre-OST0000 [12701.241619] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12710.996347] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12712.576219] LustreError: 393198:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12712.597907] LustreError: 393198:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12712.608984] LustreError: 393198:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12716.942416] LustreError: 393246:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12716.951910] LustreError: 393246:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12721.990572] Lustre: server umount lustre-OST0000 complete [12725.339124] LustreError: 393496:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12725.362967] LustreError: 393496:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12728.802851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12728.806056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12728.824487] Lustre: Skipped 1 previous similar message [12728.829894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12728.836739] Lustre: Skipped 1 previous similar message [12729.826829] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12729.845849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12729.860512] Lustre: Skipped 1 previous similar message [12734.944523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12734.952513] Lustre: Skipped 1 previous similar message [12738.095591] Lustre: server umount lustre-MDT0000 complete [12740.065595] LustreError: 390208:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12740.083249] LustreError: 390208:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12742.883181] LustreError: 390189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789075879 with bad export cookie 15337048574270889999 [12742.884566] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12742.901403] LustreError: 390189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12752.634140] Lustre: server umount lustre-MDT0001 complete [12763.195969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12768.419641] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12771.306697] LustreError: 394657:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12779.003753] Lustre: 394996:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.102@tcp [12782.032828] Lustre: server umount MGS complete [12796.336049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12797.009394] LustreError: 395723:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12797.040608] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12797.049863] Lustre: Skipped 1 previous similar message [12797.146979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12804.196350] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12813.562401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12813.852782] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12813.964641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12819.125682] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12828.088902] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12835.795420] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12839.640414] Lustre: Permanently reactivating lustre-OST0000 [12839.699888] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12839.714486] Lustre: Skipped 2 previous similar messages [12870.112641] LustreError: 397504:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12879.829527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12880.080395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12886.687340] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12896.660188] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12903.764626] Lustre: server umount lustre-OST0000 complete [12911.076251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12911.080069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12911.081601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12911.088815] Lustre: Skipped 1 previous similar message [12919.406780] Lustre: server umount lustre-MDT0000 complete [12920.357260] Lustre: 387012:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789076041/real 1789076041] req@000000007e723dac x1875981814043136/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789076057 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12920.378706] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12920.386759] LustreError: 397405:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12920.397445] LustreError: 397405:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12921.261655] Lustre: 387010:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789076042/real 1789076042] req@0000000035ddbbfe x1875981814043904/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789076058 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12923.835778] LustreError: 396457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789076060 with bad export cookie 15337048574270891049 [12923.844908] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12923.847862] LustreError: 396457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12924.138730] Lustre: server umount lustre-MDT0001 complete [12940.145438] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [12944.368767] Key type lgssc unregistered [12944.788683] LNet: 400041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12944.827157] LNetError: 400041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12944.859616] LNet: Removed LNI 192.168.202.102@tcp [12946.204200] Key type .llcrypt unregistered [12946.208362] Key type ._llcrypt unregistered [12971.460348] Key type ._llcrypt registered [12971.476628] Key type .llcrypt registered [12971.690156] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [12990.633180] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [12991.483809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12991.688755] alg: No test for adler32 (adler32-zlib) [12992.900381] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [12993.202483] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [12994.887678] Key type lgssc registered [12996.650966] Lustre: Echo OBD driver; http://www.lustre.org/ [13043.978553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13044.006912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13045.303683] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13045.324139] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13045.386258] Lustre: lustre-MDT0000: new disk, initializing [13045.436506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13045.454191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13051.061778] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13061.703879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13061.858213] Lustre: 403199:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13061.890603] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13061.894456] Lustre: Skipped 1 previous similar message [13061.976583] Lustre: lustre-MDT0001: new disk, initializing [13062.093397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13062.148613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13062.169168] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13067.495905] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13082.739156] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13092.319507] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13102.314470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13102.732287] Lustre: lustre-OST0000: new disk, initializing [13102.738738] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13102.746097] Lustre: 405169:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13102.844049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13109.385817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13109.403152] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13109.551934] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13110.621395] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13120.739479] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13125.897877] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13126.071099] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13131.730300] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13132.058847] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13135.328852] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13135.331101] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13135.338883] LustreError: Skipped 1 previous similar message [13135.343214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13135.349317] Lustre: Skipped 1 previous similar message [13135.377820] Lustre: Skipped 1 previous similar message [13140.451529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13140.478107] Lustre: Skipped 1 previous similar message [13145.569000] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13145.584339] Lustre: Skipped 1 previous similar message [13148.127683] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13148.343479] Lustre: server umount lustre-OST0000 complete [13154.272537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13154.275218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13154.288766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13155.809156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13159.175841] Lustre: server umount lustre-MDT0000 complete [13162.355522] LustreError: 403190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789076299 with bad export cookie 3915963634376792981 [13162.367500] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13162.374663] LustreError: 403190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13162.662041] Lustre: server umount lustre-MDT0001 complete [13172.845316] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [13175.051441] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 17:38:30 (1789076310) [13182.948650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13183.309995] LustreError: 407633:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13183.321323] LustreError: 407633:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [13183.357683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13188.283132] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13188.588657] LustreError: 407634:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13193.704090] LustreError: 407633:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13197.942880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13198.230979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13202.895332] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13210.560431] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13217.212943] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13225.595490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13225.971351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13233.250317] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13241.321208] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13243.308092] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [13247.320133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [13253.603710] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13253.607165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13253.638667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13253.646018] Lustre: Skipped 3 previous similar messages [13258.416172] Lustre: server umount lustre-OST0000 complete [13263.851787] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13263.879944] Lustre: Skipped 1 previous similar message [13263.903575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13263.919420] Lustre: Skipped 1 previous similar message [13269.956526] Lustre: server umount lustre-MDT0000 complete [13273.060344] LustreError: 407634:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13273.087849] LustreError: 407634:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13274.599925] LustreError: 407613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789076411 with bad export cookie 3915963634376793870 [13274.600739] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13274.617795] LustreError: 407613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13274.863407] Lustre: server umount lustre-MDT0001 complete [13285.406071] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [13289.015949] Key type lgssc unregistered [13289.461915] LNet: 411361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13289.468505] LNetError: 411361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13289.493298] LNet: Removed LNI 192.168.202.102@tcp [13290.529978] Key type .llcrypt unregistered [13290.532353] Key type ._llcrypt unregistered [13322.321684] Key type ._llcrypt registered [13322.333140] Key type .llcrypt registered [13322.603717] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [13322.902290] Key type .llcrypt unregistered [13322.904024] Key type ._llcrypt unregistered [13345.691218] Key type ._llcrypt registered [13345.692744] Key type .llcrypt registered [13345.899895] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [13347.017267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13347.191146] alg: No test for adler32 (adler32-zlib) [13348.535795] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [13348.953542] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [13350.681159] Key type lgssc registered [13352.012604] Lustre: Echo OBD driver; http://www.lustre.org/ [13359.198599] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13359.240693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13360.645220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13365.616864] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13370.852960] LustreError: 413570:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13370.887615] LustreError: 413570:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13375.646922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13375.970891] LustreError: 413569:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13376.010539] LustreError: 413569:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13376.022128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13381.096775] LustreError: 413570:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13381.131423] LustreError: 413570:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13383.949513] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13385.193262] LustreError: 413569:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13385.221821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13385.228212] LustreError: 413569:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13390.313071] LustreError: 413566:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13390.321812] LustreError: 413566:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13390.970896] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13397.493860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13398.015153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13401.333710] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13410.783930] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13410.791555] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13410.802592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13413.344975] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13413.357149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13413.359350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13413.392372] Lustre: Skipped 1 previous similar message [13415.848193] Lustre: server umount lustre-OST0000 complete [13423.591168] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13423.599852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13423.611139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13423.621056] Lustre: Skipped 1 previous similar message [13425.948225] Lustre: server umount lustre-MDT0000 complete [13428.733850] LustreError: 414514:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13428.786878] LustreError: 414514:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [13432.067575] LustreError: 413550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789076569 with bad export cookie 14399054485962254453 [13432.203940] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13432.284560] LustreError: 413550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13432.545743] Lustre: server umount lustre-MDT0001 complete [13447.692636] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [13450.780092] Key type lgssc unregistered [13451.111255] LNet: 416717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13451.123537] LNetError: 416717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13451.152868] LNet: Removed LNI 192.168.202.102@tcp [13452.470952] Key type .llcrypt unregistered [13452.473348] Key type ._llcrypt unregistered [13480.858625] Key type ._llcrypt registered [13480.860412] Key type .llcrypt registered [13480.971605] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [13481.630106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13481.774369] alg: No test for adler32 (adler32-zlib) [13483.031314] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [13483.271477] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [13484.959312] Key type lgssc registered [13486.362343] Lustre: Echo OBD driver; http://www.lustre.org/ [13494.785734] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13494.801940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13510.563343] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13515.745802] LustreError: 418104:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.102@tcp: failed processing log, type 4: rc = -110 [13542.369428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13551.906476] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13561.584795] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13567.826573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13568.751953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13568.882948] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13574.115407] LustreError: 418129:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13574.144564] LustreError: 418129:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13575.235637] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13577.188656] LustreError: 419142:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13577.210697] LustreError: 419142:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13582.307119] LustreError: 419142:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13582.340285] LustreError: 419142:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13585.808816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13586.422330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13591.687919] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13599.354785] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13606.737208] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13614.846212] Lustre: server umount lustre-OST0000 complete [13621.738384] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13621.752449] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13621.769617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13627.363164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13627.370047] Lustre: Skipped 3 previous similar messages [13629.569750] Lustre: server umount lustre-MDT0000 complete [13632.488682] LustreError: 419877:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13632.516751] LustreError: 419877:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13633.503374] Lustre: 417439:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789076754/real 1789076754] req@000000003b0fdef0 x1875982805925248/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789076770 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13633.523687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13633.544499] Lustre: Skipped 1 previous similar message [13634.192270] LustreError: 418111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789076771 with bad export cookie 17085945125354949031 [13634.208955] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13634.214656] LustreError: 418111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13634.665225] Lustre: server umount lustre-MDT0001 complete [13646.826350] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [13650.312901] Key type lgssc unregistered [13650.756668] LNet: 421673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13650.759529] LNetError: 421673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13650.785420] LNet: Removed LNI 192.168.202.102@tcp [13652.014413] Key type .llcrypt unregistered [13652.017826] Key type ._llcrypt unregistered [13680.134348] Key type ._llcrypt registered [13680.136271] Key type .llcrypt registered [13680.263651] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [13681.670321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13681.804644] alg: No test for adler32 (adler32-zlib) [13683.556813] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [13684.329082] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [13686.079272] Key type lgssc registered [13687.168869] Lustre: Echo OBD driver; http://www.lustre.org/ [13695.284322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13695.293908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13696.699715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13704.275687] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13706.216590] LustreError: 423093:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13706.233280] LustreError: 423093:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13711.336506] LustreError: 423092:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13714.844047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13715.270290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13721.724980] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13730.289338] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13737.250461] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13746.399415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13746.959333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13755.475869] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13765.533714] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13766.549288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13771.745689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13771.758513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13771.772596] Lustre: Skipped 1 previous similar message [13771.798486] Lustre: Skipped 2 previous similar messages [13773.529184] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.2@tcp (stopping) [13776.868154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13776.879331] Lustre: Skipped 2 previous similar messages [13777.343119] Lustre: server umount lustre-MDT0000 complete [13778.570821] LustreError: 424382:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13780.327630] LustreError: 423074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.2@tcp arrived at 1789076917 with bad export cookie 9680618772567381105 [13780.374551] LustreError: 423074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13781.990028] LustreError: 424018:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13782.012940] LustreError: 424018:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13782.038828] LustreError: 423073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789076919 with bad export cookie 9680618772567380209 [13782.045860] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13787.104738] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13787.108628] LustreError: 423093:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13787.119887] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13787.119895] Lustre: Skipped 1 previous similar message [13787.123425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13787.140872] LustreError: 423093:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13788.419445] Lustre: server umount lustre-OST0000 complete [13796.976299] Lustre: server umount lustre-MDT0001 complete [13810.750816] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [13814.097721] Key type lgssc unregistered [13814.288840] LNet: 426779:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13814.297408] LNetError: 426779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13814.319589] LNet: Removed LNI 192.168.202.102@tcp [13815.311053] Key type .llcrypt unregistered [13815.312851] Key type ._llcrypt unregistered [13842.760648] Key type ._llcrypt registered [13842.763669] Key type .llcrypt registered [13842.907451] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [13844.014908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13844.197506] alg: No test for adler32 (adler32-zlib) [13845.288625] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [13845.785879] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [13847.447223] Key type lgssc registered [13848.695858] Lustre: Echo OBD driver; http://www.lustre.org/ [13857.174802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13857.194808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13858.616574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13863.319579] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13869.032198] LustreError: 428194:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13869.050047] LustreError: 428194:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13872.305340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13872.720149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13878.171850] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13886.579912] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13892.874617] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13901.159989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13901.832896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13904.953074] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13907.296718] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13916.464105] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13922.275385] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13922.329532] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13922.340553] Lustre: Skipped 1 previous similar message [13923.497468] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.2@tcp (stopping) [13927.389494] Lustre: server umount lustre-MDT0001 complete [13927.405788] LustreError: 428191:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13927.437258] LustreError: 428191:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13928.632749] LustreError: 428189:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13932.513236] LustreError: 428190:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13932.526273] LustreError: 428190:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13937.633145] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13937.635908] LustreError: 428194:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13937.661868] Lustre: Skipped 1 previous similar message [13937.664514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13942.087669] Lustre: server umount lustre-OST0000 complete [13946.686868] Lustre: server umount lustre-MDT0000 complete [13966.467632] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [13970.765138] Key type lgssc unregistered [13971.416565] LNet: 431877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13971.425748] LNetError: 431877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13971.457408] LNet: Removed LNI 192.168.202.102@tcp [13972.818321] Key type .llcrypt unregistered [13972.822859] Key type ._llcrypt unregistered [14001.520426] Key type ._llcrypt registered [14001.522341] Key type .llcrypt registered [14001.588837] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [14020.401466] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [14021.322102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14021.391427] alg: No test for adler32 (adler32-zlib) [14022.461970] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [14022.665475] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [14024.487288] Key type lgssc registered [14025.980485] Lustre: Echo OBD driver; http://www.lustre.org/ [14070.249349] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [14086.840155] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14086.848269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14088.373733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14088.490655] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14088.610048] Lustre: lustre-MDT0000: new disk, initializing [14088.750962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14088.785834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14093.512494] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [14112.248415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14112.632885] Lustre: lustre-OST0000: new disk, initializing [14112.637408] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14112.643699] Lustre: Skipped 1 previous similar message [14112.663451] Lustre: 436928:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14112.765817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14121.395261] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [14122.121443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14122.150250] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14122.207418] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14132.121241] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14148.792891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14148.861247] Lustre: 435907:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14148.922449] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14149.022847] Lustre: lustre-MDT0001: new disk, initializing [14149.115196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14149.158304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [14149.178623] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [14153.261553] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [14166.173925] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [14166.485612] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [14172.088329] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [14172.388026] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [14182.888497] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14182.909750] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [14182.911495] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14182.920697] Lustre: Skipped 1 previous similar message [14186.664632] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.2@tcp (stopping) [14188.001880] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14188.009754] Lustre: Skipped 1 previous similar message [14188.371414] Lustre: server umount lustre-MDT0001 complete [14198.246433] LustreError: 435913:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14198.273651] LustreError: 435913:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [14203.374428] LustreError: 435914:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14203.405422] LustreError: 435914:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14208.482687] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14208.482792] LustreError: 438258:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14208.498281] Lustre: Skipped 1 previous similar message [14208.538202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14213.610507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14213.611328] LustreError: 435919:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14218.725438] LustreError: 438258:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14222.303654] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14222.695827] Lustre: server umount lustre-OST0000 complete [14234.146597] Lustre: server umount lustre-MDT0000 complete [14250.252357] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [14253.817836] Key type lgssc unregistered [14254.153668] LNet: 440677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14254.162970] LNetError: 440677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14254.195819] LNet: Removed LNI 192.168.202.102@tcp [14255.533371] Key type .llcrypt unregistered [14255.534759] Key type ._llcrypt unregistered [14306.854552] Key type ._llcrypt registered [14306.858255] Key type .llcrypt registered [14306.996888] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [14307.411641] Key type .llcrypt unregistered [14307.415048] Key type ._llcrypt unregistered [14325.170542] Key type ._llcrypt registered [14325.173098] Key type .llcrypt registered [14325.273580] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [14326.472375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14326.500649] alg: No test for adler32 (adler32-zlib) [14327.586259] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [14327.857387] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [14329.567316] Key type lgssc registered [14330.861537] Lustre: Echo OBD driver; http://www.lustre.org/ [14359.572121] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14359.612381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14361.027055] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14361.114466] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14361.385825] Lustre: lustre-MDT0000: new disk, initializing [14361.591866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14361.648299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14368.022453] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [14378.566162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14378.728214] Lustre: 444053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14378.789139] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14378.791382] Lustre: Skipped 1 previous similar message [14378.857900] Lustre: lustre-MDT0001: new disk, initializing [14378.945755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14378.981326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14378.993431] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14384.083540] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [14396.214715] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14403.178282] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14412.147430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14412.446872] Lustre: lustre-OST0000: new disk, initializing [14412.449919] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14412.456708] Lustre: 446022:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14412.576732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14418.980684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14418.992946] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14419.130069] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14419.852112] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [14429.301708] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14435.303772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14435.310941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14435.337094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14439.398746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14439.399374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14439.414746] Lustre: Skipped 1 previous similar message [14439.422409] Lustre: Skipped 1 previous similar message [14441.387116] Lustre: server umount lustre-MDT0000 complete [14444.969457] LustreError: 444044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789077581 with bad export cookie 8305375968506956051 [14444.973074] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14444.991982] LustreError: 444044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14445.568498] Lustre: server umount lustre-MDT0001 complete [14460.686930] Lustre: server umount lustre-OST0000 complete [14476.825699] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 18:00:11 (1789077611) [14486.882163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14487.604306] LustreError: 448049:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14487.614725] LustreError: 448049:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [14487.667673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14492.071302] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [14493.166523] LustreError: 448050:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14497.262788] LustreError: 448049:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14500.366566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14500.752445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14506.622651] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [14517.019113] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14524.698516] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14545.839472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14546.016362] Lustre: Found index 0 for lustre-OST0000, updating log [14546.135519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14551.876660] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [14558.466121] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14558.539208] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14568.934537] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14568.981460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14568.984699] Lustre: Skipped 1 previous similar message [14570.724678] Lustre: server umount lustre-OST0000 complete [14587.872105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14587.877302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14587.895940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14587.901057] Lustre: Skipped 1 previous similar message [14592.481037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14592.497060] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14602.207431] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14602.479970] Lustre: server umount lustre-MDT0000 complete [14602.722528] LustreError: 450806:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14602.737295] LustreError: 450806:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14603.424277] LNet: 1 local NIs in recovery (showing 1): 192.168.202.102@tcp [14607.238484] LustreError: 448029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789077744 with bad export cookie 8305375968506957451 [14607.268151] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14607.729887] Lustre: server umount lustre-MDT0001 complete [14617.227147] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 18:02:33 (1789077753) [15243.701230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15243.973898] LustreError: 454474:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15244.019045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15249.380484] LustreError: 454475:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15249.414204] LustreError: 454475:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15250.108087] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [15253.484385] LustreError: 454474:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15259.639593] LustreError: 454475:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15259.661855] LustreError: 454475:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15261.348400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15261.735301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15266.948416] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [15274.541334] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15279.545236] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15294.880929] LNet: 1 local NIs in recovery (showing 1): 192.168.202.102@tcp [15298.478382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15298.748590] Lustre: Found index 0 for lustre-OST0000, updating log [15299.055305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15306.042131] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [15313.170808] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15315.752637] Lustre: server umount lustre-OST0000 complete [15321.058498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15321.073470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15321.093960] Lustre: Skipped 1 previous similar message [15321.124926] Lustre: Skipped 4 previous similar messages [15322.144828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15322.147281] Lustre: Skipped 1 previous similar message [15324.195829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15324.199610] Lustre: Skipped 3 previous similar messages [15326.575342] Lustre: server umount lustre-MDT0000 complete [15327.271347] LustreError: 454474:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15329.537906] LustreError: 454455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789078466 with bad export cookie 8305375968506958515 [15329.546410] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15329.575181] LustreError: 454455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [15330.026441] Lustre: server umount lustre-MDT0001 complete [15354.271168] LNet: 1 local NIs in recovery (showing 1): 192.168.202.102@tcp [15362.739562] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [15374.675944] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [15413.225444] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 18:15:48 (1789078548) [15434.804654] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [15439.927153] Key type lgssc unregistered [15440.526687] LNet: 461084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15440.570097] LNetError: 461084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15562.471653] LNet: Removed LNI 192.168.202.102@tcp [15563.589142] Key type .llcrypt unregistered [15563.591576] Key type ._llcrypt unregistered [15582.623930] Key type ._llcrypt registered [15582.631800] Key type .llcrypt registered [15582.729515] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [15584.389180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15584.461443] alg: No test for adler32 (adler32-zlib) [15585.497520] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [15585.773778] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [15587.538058] Key type lgssc registered [15588.822266] Lustre: Echo OBD driver; http://www.lustre.org/ [15593.441669] LNet: Added LNI 192.168.202.102@tcp100 [8/256/0/180] [15594.350482] LNet: Added LNI 192.168.202.102@tcp101 [8/256/0/180] [15595.339858] LNet: Added LNI 192.168.202.102@tcp102 [8/256/0/180] [15596.407850] LNet: Added LNI 192.168.202.102@tcp103 [8/256/0/180] [15597.308513] LNet: Added LNI 192.168.202.102@tcp104 [8/256/0/180] [15598.337781] LNet: Added LNI 192.168.202.102@tcp105 [8/256/0/180] [15599.339631] LNet: Added LNI 192.168.202.102@tcp106 [8/256/0/180] [15600.366463] LNet: Added LNI 192.168.202.102@tcp107 [8/256/0/180] [15601.329647] LNet: Added LNI 192.168.202.102@tcp108 [8/256/0/180] [15601.985993] LNet: Added LNI 192.168.202.102@tcp109 [8/256/0/180] [15602.814642] LNet: Added LNI 192.168.202.102@tcp110 [8/256/0/180] [15603.979163] LNet: Added LNI 192.168.202.102@tcp111 [8/256/0/180] [15604.594078] LNet: Added LNI 192.168.202.102@tcp112 [8/256/0/180] [15605.228495] LNet: Added LNI 192.168.202.102@tcp113 [8/256/0/180] [15606.365457] LNet: Added LNI 192.168.202.102@tcp114 [8/256/0/180] [15607.365063] LNet: Added LNI 192.168.202.102@tcp115 [8/256/0/180] [15608.227678] LNet: Added LNI 192.168.202.102@tcp116 [8/256/0/180] [15609.035337] LNet: Added LNI 192.168.202.102@tcp117 [8/256/0/180] [15610.103839] LNet: Added LNI 192.168.202.102@tcp118 [8/256/0/180] [15610.656369] LNet: Added LNI 192.168.202.102@tcp119 [8/256/0/180] [15611.385816] LNet: Added LNI 192.168.202.102@tcp120 [8/256/0/180] [15612.264424] LNet: Added LNI 192.168.202.102@tcp121 [8/256/0/180] [15612.831687] LNet: Added LNI 192.168.202.102@tcp122 [8/256/0/180] [15613.900491] LNet: Added LNI 192.168.202.102@tcp123 [8/256/0/180] [15614.802940] LNet: Added LNI 192.168.202.102@tcp124 [8/256/0/180] [15616.264889] LNet: Added LNI 192.168.202.102@tcp125 [8/256/0/180] [15617.241364] LNet: Added LNI 192.168.202.102@tcp126 [8/256/0/180] [15618.021684] LNet: Added LNI 192.168.202.102@tcp127 [8/256/0/180] [15618.913221] LNet: Added LNI 192.168.202.102@tcp128 [8/256/0/180] [15619.902630] LNet: Added LNI 192.168.202.102@tcp129 [8/256/0/180] [15620.805849] LNet: Added LNI 192.168.202.102@tcp130 [8/256/0/180] [15621.790993] LNet: Added LNI 192.168.202.102@tcp131 [8/256/0/180] [15623.053706] LNet: Added LNI 192.168.202.102@tcp132 [8/256/0/180] [15623.865577] LNet: Added LNI 192.168.202.102@tcp133 [8/256/0/180] [15624.526564] LNet: Added LNI 192.168.202.102@tcp134 [8/256/0/180] [15625.449622] LNet: Added LNI 192.168.202.102@tcp135 [8/256/0/180] [15633.054982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15633.106066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15634.289632] Lustre: 464114:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.102@tcp: can't add NID 192.168.202.102@tcp130: rc = -75 [15634.381572] LNetError: 461575:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.102 [15634.389324] LNetError: Protocol error connecting to 192.168.202.102@tcp120 at host 192.168.202.102:988: is it running a compatible version of Lustre? [15634.532061] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15634.577682] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15634.671504] Lustre: lustre-MDT0000: new disk, initializing [15634.770859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15634.793499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15634.975476] LNet: 5 local NIs in recovery (showing 5): 192.168.202.102@tcp120, 192.168.202.102@tcp123, 192.168.202.102@tcp124, 192.168.202.102@tcp128, 192.168.202.102@tcp129 [15635.425698] LNetError: 464139:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.102 [15635.454574] LNetError: 464139:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [15635.458995] LNetError: Protocol error connecting to 192.168.202.102@tcp101 at host 192.168.202.102:988: is it running a compatible version of Lustre? [15635.472839] LNetError: Skipped 10 previous similar messages [15638.112935] LNetError: 464142:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.102 [15638.117802] LNetError: 464142:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [15638.123577] LNetError: Protocol error connecting to 192.168.202.102@tcp124 at host 192.168.202.102:988: is it running a compatible version of Lustre? [15638.140128] LNetError: Skipped 5 previous similar messages [15639.980765] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [15642.210689] LNetError: 464134:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.102 [15642.222854] LNetError: Protocol error connecting to 192.168.202.102@tcp124 at host 192.168.202.102:988: is it running a compatible version of Lustre? [15650.407655] LNetError: 464135:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.102 [15650.428802] LNetError: Protocol error connecting to 192.168.202.102@tcp124 at host 192.168.202.102:988: is it running a compatible version of Lustre? [15650.863377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15651.065726] Lustre: 464129:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15651.164744] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15651.169556] Lustre: Skipped 1 previous similar message [15651.278988] Lustre: lustre-MDT0001: new disk, initializing [15651.331768] Lustre: 464934:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.102@tcp: can't add NID 192.168.202.102@tcp130: rc = -75 [15651.342727] Lustre: 464934:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15651.503062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15651.556300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15651.571073] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15656.711484] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [15657.554968] Lustre: 465221:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.102@tcp: can't add NID 192.168.202.102@tcp130: rc = -75 [15657.563122] Lustre: 465221:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15665.638502] LNetError: 461573:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.102 [15665.656394] LNetError: Protocol error connecting to 192.168.202.102@tcp121 at host 192.168.202.102:988: is it running a compatible version of Lustre? [15666.709403] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15673.122358] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15679.983121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15680.182294] Lustre: lustre-OST0000: new disk, initializing [15680.185202] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15680.190390] Lustre: 466105:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15680.266855] Lustre: 466110:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.102@tcp: can't add NID 192.168.202.102@tcp130: rc = -75 [15680.270185] Lustre: 466110:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15680.312268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15682.032874] LNetError: 464145:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.102 [15682.051788] LNetError: 464145:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [15682.062665] LNetError: Protocol error connecting to 192.168.202.102@tcp114 at host 192.168.202.102:988: is it running a compatible version of Lustre? [15682.074019] LNetError: Skipped 11 previous similar messages [15686.080413] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [15689.264525] Lustre: 466579:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.102@tcp: can't add NID 192.168.202.102@tcp130: rc = -75 [15689.272347] Lustre: 466579:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15689.342619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15689.352687] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15689.531634] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15694.431172] LNet: 1 local NIs in recovery (showing 1): 192.168.202.102@tcp124 [15695.443821] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15705.057911] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15705.062309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15705.077209] Lustre: Skipped 1 previous similar message [15705.083722] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15705.093696] Lustre: Skipped 1 previous similar message [15710.185452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15710.200358] Lustre: Skipped 1 previous similar message [15715.300854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15715.310343] Lustre: Skipped 1 previous similar message [15717.343131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15717.691281] Lustre: server umount lustre-OST0000 complete [15722.977307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15722.985688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15723.000961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15725.557459] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15729.120142] LNetError: 461575:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.102 [15729.130618] LNetError: 461575:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 19 previous similar messages [15729.144622] LNetError: Protocol error connecting to 192.168.202.102@tcp121 at host 192.168.202.102:988: is it running a compatible version of Lustre? [15729.161396] LNetError: Skipped 19 previous similar messages [15730.659160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15730.662099] Lustre: Skipped 2 previous similar messages [15736.288878] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15736.589417] Lustre: server umount lustre-MDT0000 complete [15740.862740] LustreError: 464933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789078877 with bad export cookie 8494432013233398049 [15740.866435] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15740.873724] LustreError: 464933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15741.514480] Lustre: server umount lustre-MDT0001 complete [15752.805229] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [15755.628989] Key type lgssc unregistered [15756.053239] LNet: 468021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15756.067058] LNetError: 468021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15756.079164] LNet: Removed LNI 192.168.202.102@tcp [15757.087810] LNet: Removed LNI 192.168.202.102@tcp100 [15757.095370] LNetError: 468021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15757.099057] LNetError: 468021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15757.103571] LNet: Removed LNI 192.168.202.102@tcp101 [15758.112110] LNet: Removed LNI 192.168.202.102@tcp102 [15758.125063] LNetError: 468021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15758.134409] LNetError: 468021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15758.144148] LNet: Removed LNI 192.168.202.102@tcp103 [15759.199933] LNet: Removed LNI 192.168.202.102@tcp104 [15760.224132] LNet: Removed LNI 192.168.202.102@tcp105 [15760.239757] LNetError: 468021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15760.255776] LNetError: 468021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15760.280653] LNet: Removed LNI 192.168.202.102@tcp106 [15761.311290] LNet: Removed LNI 192.168.202.102@tcp107 [15762.336331] LNet: Removed LNI 192.168.202.102@tcp108 [15763.361559] LNet: Removed LNI 192.168.202.102@tcp109 [15764.383455] LNet: Removed LNI 192.168.202.102@tcp110 [15764.388811] LNetError: 468021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15764.395354] LNetError: 468021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15764.408140] LNet: Removed LNI 192.168.202.102@tcp111 [15765.471394] LNet: Removed LNI 192.168.202.102@tcp112 [15766.496639] LNet: Removed LNI 192.168.202.102@tcp113 [15767.519441] LNet: Removed LNI 192.168.202.102@tcp114 [15768.543500] LNet: Removed LNI 192.168.202.102@tcp115 [15769.568054] LNet: Removed LNI 192.168.202.102@tcp116 [15770.591203] LNet: Removed LNI 192.168.202.102@tcp117 [15771.617977] LNet: Removed LNI 192.168.202.102@tcp118 [15772.704107] LNet: Removed LNI 192.168.202.102@tcp119 [15772.713132] LNetError: 468021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15772.721309] LNetError: 468021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15772.733173] LNet: Removed LNI 192.168.202.102@tcp120 [15773.792559] LNet: Removed LNI 192.168.202.102@tcp121 [15774.886941] LNet: Removed LNI 192.168.202.102@tcp122 [15775.905162] LNet: Removed LNI 192.168.202.102@tcp123 [15775.911108] LNet: Removed LNI 192.168.202.102@tcp124 [15776.927209] LNet: Removed LNI 192.168.202.102@tcp125 [15777.951714] LNet: Removed LNI 192.168.202.102@tcp126 [15778.975161] LNet: Removed LNI 192.168.202.102@tcp127 [15779.999542] LNet: Removed LNI 192.168.202.102@tcp128 [15781.023390] LNet: Removed LNI 192.168.202.102@tcp129 [15781.027444] LNet: Removed LNI 192.168.202.102@tcp130 [15781.034283] LNet: Removed LNI 192.168.202.102@tcp131 [15781.039394] LNet: Removed LNI 192.168.202.102@tcp132 [15781.043727] LNet: Removed LNI 192.168.202.102@tcp133 [15781.056270] LNet: Removed LNI 192.168.202.102@tcp134 [15781.083045] LNet: Removed LNI 192.168.202.102@tcp135 [15782.177156] Key type .llcrypt unregistered [15782.179650] Key type ._llcrypt unregistered [15809.753695] Key type ._llcrypt registered [15809.759435] Key type .llcrypt registered [15809.889623] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [15826.448939] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [15827.928506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15828.007915] alg: No test for adler32 (adler32-zlib) [15829.221410] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [15829.680463] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [15831.407553] Key type lgssc registered [15832.781444] Lustre: Echo OBD driver; http://www.lustre.org/ [15868.501135] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 18:23:24 (1789079004) [15885.712758] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [15888.656816] Key type lgssc unregistered [15889.060578] LNet: 472108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15889.070129] LNetError: 472108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15889.088491] LNet: Removed LNI 192.168.202.102@tcp [15890.004611] Key type .llcrypt unregistered [15890.006659] Key type ._llcrypt unregistered [15910.953499] Key type ._llcrypt registered [15910.955362] Key type .llcrypt registered [15911.033672] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [15912.202623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15912.309589] alg: No test for adler32 (adler32-zlib) [15913.352956] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [15913.565049] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [15915.207221] Key type lgssc registered [15916.158152] Lustre: Echo OBD driver; http://www.lustre.org/ [15926.221026] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15926.251360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15927.635444] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15927.652088] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15927.709514] Lustre: lustre-MDT0000: new disk, initializing [15927.766945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15927.779485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15932.427296] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [15943.506544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15943.670667] Lustre: 473315:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15943.847814] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15943.852654] Lustre: Skipped 1 previous similar message [15943.910438] Lustre: lustre-MDT0001: new disk, initializing [15944.027563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15944.057018] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15944.071317] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15948.519103] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [15958.119428] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15963.601115] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15970.354614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15970.511957] Lustre: lustre-OST0000: new disk, initializing [15970.515118] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15970.518887] Lustre: 475286:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15970.594855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15977.388761] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [15980.631722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15980.644304] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15980.812148] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15986.896841] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15990.577542] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing lnet_if_list [15996.630542] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15996.650993] LNet: Added LNI 192.168.202.102@tcp10 [8/256/0/180] [16001.824571] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [16001.860336] LNet: Added LNI 192.168.202.102@tcp11 [8/256/0/180] [16005.998731] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [16006.021743] LNet: Added LNI 192.168.202.102@tcp12 [8/256/0/180] [16009.914715] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [16009.941298] LNet: Added LNI 192.168.202.102@tcp13 [8/256/0/180] [16014.266747] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [16014.296299] LNet: Added LNI 192.168.202.102@tcp14 [8/256/0/180] [16018.412713] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [16018.449811] LNet: Added LNI 192.168.202.102@tcp15 [8/256/0/180] [16022.356742] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [16022.392973] LNet: Added LNI 192.168.202.102@tcp16 [8/256/0/180] [16027.289693] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [16027.336194] LNet: Added LNI 192.168.202.102@tcp17 [8/256/0/180] [16031.925916] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [16031.954617] LNet: Added LNI 192.168.202.102@tcp18 [8/256/0/180] [16037.351884] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [16037.416590] LNet: Added LNI 192.168.202.102@tcp19 [8/256/0/180] [16042.656613] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [16042.683104] LNet: Added LNI 192.168.202.102@tcp20 [8/256/0/180] [16048.141575] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [16048.172438] LNet: Added LNI 192.168.202.102@tcp21 [8/256/0/180] [16054.521855] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [16054.583660] LNet: Added LNI 192.168.202.102@tcp22 [8/256/0/180] [16060.880752] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [16060.934767] LNet: Added LNI 192.168.202.102@tcp23 [8/256/0/180] [16067.135561] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [16067.182756] LNet: Added LNI 192.168.202.102@tcp24 [8/256/0/180] [16073.247524] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [16073.289354] LNet: Added LNI 192.168.202.102@tcp25 [8/256/0/180] [16078.961547] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [16079.012628] LNet: Added LNI 192.168.202.102@tcp26 [8/256/0/180] [16085.110336] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [16085.128654] LNet: Added LNI 192.168.202.102@tcp27 [8/256/0/180] [16092.777976] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [16092.820357] LNet: Added LNI 192.168.202.102@tcp28 [8/256/0/180] [16100.024790] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [16100.071759] LNet: Added LNI 192.168.202.102@tcp29 [8/256/0/180] [16107.548254] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [16107.581351] LNet: Added LNI 192.168.202.102@tcp30 [8/256/0/180] [16113.199197] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [16113.242885] LNet: Added LNI 192.168.202.102@tcp31 [8/256/0/180] [16118.103604] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [16118.127863] LNet: Added LNI 192.168.202.102@tcp32 [8/256/0/180] [16123.677979] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [16123.710316] LNet: Added LNI 192.168.202.102@tcp33 [8/256/0/180] [16129.781316] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [16129.810875] LNet: Added LNI 192.168.202.102@tcp34 [8/256/0/180] [16135.947721] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [16135.979609] LNet: Added LNI 192.168.202.102@tcp35 [8/256/0/180] [16142.657148] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [16142.679351] LNet: Added LNI 192.168.202.102@tcp36 [8/256/0/180] [16149.295658] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [16149.320644] LNet: Added LNI 192.168.202.102@tcp37 [8/256/0/180] [16156.040438] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [16156.061467] LNet: Added LNI 192.168.202.102@tcp38 [8/256/0/180] [16163.999715] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [16164.041775] LNet: Added LNI 192.168.202.102@tcp39 [8/256/0/180] [16171.355161] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [16171.377668] LNet: Added LNI 192.168.202.102@tcp40 [8/256/0/180] [16177.667256] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [16177.694000] LNet: Added LNI 192.168.202.102@tcp41 [8/256/0/180] [16185.101364] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [16185.145300] LNet: Added LNI 192.168.202.102@tcp42 [8/256/0/180] [16193.353175] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [16193.374873] LNet: Added LNI 192.168.202.102@tcp43 [8/256/0/180] [16201.652807] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [16201.682490] LNet: Added LNI 192.168.202.102@tcp44 [8/256/0/180] [16209.805752] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [16209.859725] LNet: Added LNI 192.168.202.102@tcp45 [8/256/0/180] [16217.545846] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [16217.565926] LNet: Added LNI 192.168.202.102@tcp46 [8/256/0/180] [16226.880597] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [16226.920681] LNet: Added LNI 192.168.202.102@tcp47 [8/256/0/180] [16237.566448] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [16237.610134] LNet: Added LNI 192.168.202.102@tcp48 [8/256/0/180] [16247.034757] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [16247.068647] LNet: Added LNI 192.168.202.102@tcp49 [8/256/0/180] [16256.945715] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [16257.000107] LNet: Added LNI 192.168.202.102@tcp50 [8/256/0/180] [16276.011964] Lustre: server umount lustre-MDT0000 complete [16276.961224] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16276.976590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16277.989094] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16278.005544] Lustre: Skipped 1 previous similar message [16279.897197] LustreError: 473310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789079416 with bad export cookie 16595391820935622615 [16279.903637] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16279.911385] LustreError: 473310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16280.212749] Lustre: server umount lustre-MDT0001 complete [16295.520917] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [16299.490383] Lustre: 472588:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789079420/real 1789079420] req@0000000002a9ebc0 x1875985354101504/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789079436 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16299.530370] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16300.512117] Lustre: 472589:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789079421/real 1789079421] req@0000000081971b13 x1875985354101632/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789079437 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16303.545444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16304.607213] Lustre: 472590:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789079425/real 1789079425] req@000000003188f316 x1875985354101760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789079441 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16304.692959] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe64eacef5b4a7a0a [16304.731813] Lustre: MGC192.168.202.102@tcp: Connection restored to 0@lo (at 0@lo) [16305.054469] LustreError: 475279:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16305.084665] LustreError: 475279:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [16305.131881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16309.220131] LustreError: 475280:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16309.235046] LustreError: 475280:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [16310.240178] Lustre: 472588:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789079431/real 1789079431] req@00000000a2744cc1 x1875985354102144/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789079447 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16310.265783] Lustre: 472588:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [16310.352525] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [16314.337545] Lustre: 472589:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789079435/real 1789079435] req@00000000e006175c x1875985354102400/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789079451 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16314.339599] LustreError: 475279:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16314.419583] LustreError: 475279:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [16318.444484] LustreError: 475280:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16318.447101] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16318.457705] LustreError: 475280:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [16320.061674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16320.650529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16324.873488] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [16326.119436] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16331.944957] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16337.165205] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16361.441477] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16361.450206] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16361.453722] LustreError: Skipped 1 previous similar message [16361.457673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16361.472039] Lustre: Skipped 1 previous similar message [16361.489462] Lustre: Skipped 1 previous similar message [16367.077489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16367.086924] Lustre: Skipped 1 previous similar message [16371.167153] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16371.408848] Lustre: server umount lustre-OST0000 complete [16376.800448] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16376.808180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16376.836519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16381.374544] Lustre: server umount lustre-MDT0000 complete [16382.437267] LustreError: 499506:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16382.455482] LustreError: 499506:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [16384.888043] LustreError: 474113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789079521 with bad export cookie 16595391820935625226 [16384.897859] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16385.324793] Lustre: server umount lustre-MDT0001 complete [16395.007597] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [16397.706461] Key type lgssc unregistered [16398.046458] LNet: 502302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16398.053899] LNetError: 502302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16398.060798] LNet: Removed LNI 192.168.202.102@tcp [16398.063339] LNet: Removed LNI 192.168.202.102@tcp10 [16398.066630] LNet: Removed LNI 192.168.202.102@tcp11 [16398.073684] LNet: Removed LNI 192.168.202.102@tcp12 [16398.076578] LNet: Removed LNI 192.168.202.102@tcp13 [16398.083426] LNet: Removed LNI 192.168.202.102@tcp14 [16398.087304] LNet: Removed LNI 192.168.202.102@tcp15 [16398.090511] LNet: Removed LNI 192.168.202.102@tcp16 [16398.095233] LNet: Removed LNI 192.168.202.102@tcp17 [16398.098426] LNet: Removed LNI 192.168.202.102@tcp18 [16398.101244] LNet: Removed LNI 192.168.202.102@tcp19 [16398.105405] LNet: Removed LNI 192.168.202.102@tcp20 [16398.109993] LNet: Removed LNI 192.168.202.102@tcp21 [16398.114775] LNet: Removed LNI 192.168.202.102@tcp22 [16398.118866] LNet: Removed LNI 192.168.202.102@tcp23 [16398.123267] LNet: Removed LNI 192.168.202.102@tcp24 [16398.127722] LNet: Removed LNI 192.168.202.102@tcp25 [16398.130808] LNet: Removed LNI 192.168.202.102@tcp26 [16398.133745] LNet: Removed LNI 192.168.202.102@tcp27 [16398.136727] LNet: Removed LNI 192.168.202.102@tcp28 [16398.140283] LNet: Removed LNI 192.168.202.102@tcp29 [16398.144388] LNet: Removed LNI 192.168.202.102@tcp30 [16398.146974] LNet: Removed LNI 192.168.202.102@tcp31 [16398.152471] LNet: Removed LNI 192.168.202.102@tcp32 [16398.161231] LNet: Removed LNI 192.168.202.102@tcp33 [16398.165479] LNet: Removed LNI 192.168.202.102@tcp34 [16398.173400] LNet: Removed LNI 192.168.202.102@tcp35 [16398.178402] LNet: Removed LNI 192.168.202.102@tcp36 [16398.181077] LNet: Removed LNI 192.168.202.102@tcp37 [16398.185476] LNet: Removed LNI 192.168.202.102@tcp38 [16398.192178] LNet: Removed LNI 192.168.202.102@tcp39 [16398.197215] LNet: Removed LNI 192.168.202.102@tcp40 [16398.200760] LNet: Removed LNI 192.168.202.102@tcp41 [16398.204553] LNet: Removed LNI 192.168.202.102@tcp42 [16398.207542] LNet: Removed LNI 192.168.202.102@tcp43 [16398.211244] LNet: Removed LNI 192.168.202.102@tcp44 [16398.214184] LNet: Removed LNI 192.168.202.102@tcp45 [16398.217483] LNet: Removed LNI 192.168.202.102@tcp46 [16398.220715] LNet: Removed LNI 192.168.202.102@tcp47 [16398.223757] LNet: Removed LNI 192.168.202.102@tcp48 [16398.225608] LNet: Removed LNI 192.168.202.102@tcp49 [16398.236246] LNet: Removed LNI 192.168.202.102@tcp50 [16398.881781] Key type .llcrypt unregistered [16398.884083] Key type ._llcrypt unregistered [16417.902857] Key type ._llcrypt registered [16417.904587] Key type .llcrypt registered [16417.988628] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [16432.848034] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [16434.257456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16434.278550] alg: No test for adler32 (adler32-zlib) [16435.343837] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [16435.546969] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [16437.183774] Key type lgssc registered [16438.306156] Lustre: Echo OBD driver; http://www.lustre.org/ [16469.842907] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16469.858253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16471.074690] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16471.093278] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16471.169395] Lustre: lustre-MDT0000: new disk, initializing [16471.215328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16471.228374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16475.452811] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [16485.518106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16485.579384] Lustre: 505495:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16485.604416] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16485.612099] Lustre: Skipped 1 previous similar message [16485.667114] Lustre: lustre-MDT0001: new disk, initializing [16485.762682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16485.803909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16485.816554] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16490.073737] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [16498.297811] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16503.585175] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16510.710971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16510.916276] Lustre: *** cfs_fail_loc=90f, val=0*** [16510.969231] Lustre: lustre-OST0000: new disk, initializing [16510.973659] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16510.981472] Lustre: 507513:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16511.077878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16517.278853] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [16520.285061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16520.295670] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16520.474455] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16526.784051] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16536.035247] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16536.038875] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16536.057415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16536.065310] Lustre: Skipped 1 previous similar message [16536.092694] Lustre: Skipped 1 previous similar message [16541.174916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16541.178062] Lustre: Skipped 1 previous similar message [16546.280857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16546.302606] Lustre: Skipped 1 previous similar message [16548.319138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16548.479408] Lustre: server umount lustre-OST0000 complete [16552.418289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16552.424329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16552.443314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16556.516602] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16556.518800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16556.530138] Lustre: Skipped 1 previous similar message [16558.636154] Lustre: server umount lustre-MDT0000 complete [16561.947493] LustreError: 506292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789079698 with bad export cookie 7522492046427691999 [16561.958709] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16561.963534] LustreError: 506292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16562.404603] Lustre: server umount lustre-MDT0001 complete [16571.746240] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [16574.496264] Key type lgssc unregistered [16574.744820] LNet: 509418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16574.755783] LNetError: 509418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16574.782633] LNet: Removed LNI 192.168.202.102@tcp [16575.671614] Key type .llcrypt unregistered [16575.673826] Key type ._llcrypt unregistered [16592.803441] Key type ._llcrypt registered [16592.806805] Key type .llcrypt registered [16592.925473] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [16594.306802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16594.349609] alg: No test for adler32 (adler32-zlib) [16595.384681] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [16595.636786] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [16597.319332] Key type lgssc registered [16598.180650] Lustre: Echo OBD driver; http://www.lustre.org/ [16605.681553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16605.701178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16607.090813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16611.037882] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [16617.445445] LustreError: 510664:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16617.475268] LustreError: 510664:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [16620.206452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16620.475846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16625.959135] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [16634.253323] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16641.722970] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16649.104991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16649.412758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16654.834390] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [16661.723484] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16675.038145] loop0: detected capacity change from 0 to 204800000 [16675.219979] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16675.234038] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16675.257382] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16681.937848] loop0: detected capacity change from 0 to 204800000 [16682.119215] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16682.129854] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16682.143808] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16687.158786] loop0: detected capacity change from 0 to 204800000 [16692.678840] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16692.830495] Lustre: fs15246-OST0000: new disk, initializing [16692.834701] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16692.839633] Lustre: 514262:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16692.888735] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16699.429297] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [16708.216746] loop1: detected capacity change from 0 to 204800000 [16713.226389] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16713.300574] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16713.345812] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16713.411193] Lustre: fs15246-MDT0000: new disk, initializing [16713.569720] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16713.597283] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16713.681059] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16714.720256] Lustre: 515385:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000005c9ec0cc x1875986069109248/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789079867 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16718.073093] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [16723.820943] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16723.827342] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16723.847435] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16740.338524] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16740.354897] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16744.008558] Lustre: server umount fs15246-MDT0000 complete [16745.482463] LustreError: 512505:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16750.565440] LustreError: 510661:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16751.817878] Lustre: server umount fs15246-OST0000 complete [16763.361151] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16763.368065] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16763.377628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16765.409874] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16765.418488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16765.431706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16771.040485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16771.047576] Lustre: Skipped 1 previous similar message [16776.162299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16776.168967] Lustre: Skipped 2 previous similar messages [16777.183550] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16777.442460] Lustre: server umount lustre-OST0000 complete [16781.280923] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16781.294513] Lustre: Skipped 1 previous similar message [16786.401825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16786.425053] Lustre: Skipped 3 previous similar messages [16787.121533] Lustre: server umount lustre-MDT0000 complete [16791.523073] LustreError: 510665:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16791.571854] LustreError: 510665:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [16792.304124] LustreError: 510645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789079929 with bad export cookie 11688103015182664774 [16792.311454] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16792.326327] LustreError: 510645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16792.911765] Lustre: server umount lustre-MDT0001 complete [16807.063703] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [16809.962381] Key type lgssc unregistered [16810.294053] LNet: 518481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16810.297769] LNetError: 518481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16810.319348] LNet: Removed LNI 192.168.202.102@tcp [16811.600850] Key type .llcrypt unregistered [16811.605133] Key type ._llcrypt unregistered [16828.466806] Key type ._llcrypt registered [16828.468512] Key type .llcrypt registered [16828.522360] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [16828.997469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16829.037058] alg: No test for adler32 (adler32-zlib) [16830.004908] Lustre: Lustre: Build Version: 2.17.58_39_ga794951 [16830.163030] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [16831.791315] Key type lgssc registered [16832.719404] Lustre: Echo OBD driver; http://www.lustre.org/ [16867.016546] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [16874.923158] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [16905.222485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16905.263834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16906.536537] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16906.588704] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16906.645790] Lustre: lustre-MDT0000: new disk, initializing [16906.728864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16906.754677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16912.433843] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [16924.089824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16924.221841] Lustre: 522941:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16924.283673] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16924.287637] Lustre: Skipped 1 previous similar message [16924.368730] Lustre: lustre-MDT0001: new disk, initializing [16924.478912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16924.523080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16924.542334] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16930.159664] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [16939.496856] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16946.521223] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16953.903971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16954.416621] Lustre: lustre-OST0000: new disk, initializing [16954.441275] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16954.455843] Lustre: 524907:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16954.594832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16961.797756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16961.824605] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16962.118265] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16962.132561] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [16972.296381] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16978.842449] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16979.200861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16984.279811] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16984.523668] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16987.616466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16987.634938] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16987.644759] LustreError: Skipped 1 previous similar message [16987.658162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16992.794459] Lustre: server umount lustre-OST0000 complete [16997.856600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16997.862548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16997.875185] Lustre: Skipped 2 previous similar messages [16997.920290] Lustre: Skipped 2 previous similar messages [17003.002908] Lustre: server umount lustre-MDT0000 complete [17006.925201] LustreError: 522932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789080143 with bad export cookie 10344055356499687946 [17006.936957] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17006.940354] LustreError: 522932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [17007.206363] Lustre: server umount lustre-MDT0001 complete [17033.777877] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [17045.649183] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [17081.035989] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16763 sec ========= 18:43:35 (1789080215) [17083.227426] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:43:38 (1789080218) === [17086.879171] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:43:42 (1789080222) === [17127.951539] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [17131.195579] Key type lgssc unregistered [17131.520444] LNet: 531553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17131.553178] LNetError: 531553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17131.570630] LNet: Removed LNI 192.168.202.102@tcp [17132.713740] Key type .llcrypt unregistered [17132.715326] Key type ._llcrypt unregistered