[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 471388169 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002377] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007487] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011147] LSM: Security Framework initializing [ 0.012048] Yama: becoming mindful. [ 0.013055] SELinux: Initializing. [ 0.014066] *** VALIDATE selinux *** [ 0.021780] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026407] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027148] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028093] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029100] *** VALIDATE tmpfs *** [ 0.030387] *** VALIDATE proc *** [ 0.031190] *** VALIDATE cgroup *** [ 0.032011] *** VALIDATE cgroup2 *** [ 0.033243] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035146] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037029] Spectre V2 : User space: Vulnerable [ 0.039006] Speculative Store Bypass: Vulnerable [ 0.041670] debug: unmapping init [mem 0xffffffffa9c59000-0xffffffffa9c60fff] [ 0.043170] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044654] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045024] ... version: 2 [ 0.046013] ... bit width: 48 [ 0.047014] ... generic registers: 4 [ 0.048015] ... value mask: 0000ffffffffffff [ 0.049016] ... max period: 00007fffffffffff [ 0.050016] ... fixed-purpose events: 3 [ 0.051016] ... event mask: 000000070000000f [ 0.052267] rcu: Hierarchical SRCU implementation. [ 0.054554] smp: Bringing up secondary CPUs ... [ 0.055672] x86: Booting SMP configuration: [ 0.056028] .... node #0, CPUs: #1 #2 #3 [ 0.059764] smp: Brought up 1 node, 4 CPUs [ 0.061012] smpboot: Max logical packages: 1 [ 0.062020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.140659] node 0 deferred pages initialised in 76ms [ 0.144097] devtmpfs: initialized [ 0.145224] x86/mm: Memory block size: 128MB [ 0.148952] gcov: version magic: 0x41383552 [ 0.151328] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156106] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.160269] pinctrl core: initialized pinctrl subsystem [ 0.161188] [ 0.161788] ************************************************************* [ 0.165010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167011] ** ** [ 0.169011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.172014] ** ** [ 0.176018] ** This means that this kernel is built to expose internal ** [ 0.179012] ** IOMMU data structures, which may compromise security on ** [ 0.182014] ** your system. ** [ 0.184013] ** ** [ 0.186014] ** If you see this message and you are not debugging the ** [ 0.188011] ** kernel, report this immediately to your vendor! ** [ 0.191014] ** ** [ 0.195016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.198012] ************************************************************* [ 0.201104] NET: Registered protocol family 16 [ 0.203466] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.208061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.212065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.218072] cpuidle: using governor menu [ 0.219841] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.222461] PCI: Using configuration type 1 for base access [ 0.225117] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.233102] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.235016] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.240048] cryptd: max_cpu_qlen set to 1000 [ 0.243238] ACPI: Added _OSI(Module Device) [ 0.246017] ACPI: Added _OSI(Processor Device) [ 0.248029] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.249015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.254051] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.261378] ACPI: Interpreter enabled [ 0.263052] ACPI: PM: (supports S0 S3 S4 S5) [ 0.265017] ACPI: Using IOAPIC for interrupt routing [ 0.267121] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.270377] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.278772] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.281039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.284044] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.288140] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.294523] acpiphp: Slot [2] registered [ 0.296130] acpiphp: Slot [5] registered [ 0.298131] acpiphp: Slot [6] registered [ 0.299137] acpiphp: Slot [7] registered [ 0.301137] acpiphp: Slot [8] registered [ 0.302132] acpiphp: Slot [9] registered [ 0.304126] acpiphp: Slot [10] registered [ 0.306093] acpiphp: Slot [3] registered [ 0.308113] acpiphp: Slot [4] registered [ 0.310133] acpiphp: Slot [11] registered [ 0.312120] acpiphp: Slot [12] registered [ 0.314151] acpiphp: Slot [13] registered [ 0.317135] acpiphp: Slot [14] registered [ 0.319172] acpiphp: Slot [15] registered [ 0.322138] acpiphp: Slot [16] registered [ 0.324131] acpiphp: Slot [17] registered [ 0.326188] acpiphp: Slot [18] registered [ 0.328116] acpiphp: Slot [19] registered [ 0.330125] acpiphp: Slot [20] registered [ 0.332123] acpiphp: Slot [21] registered [ 0.333113] acpiphp: Slot [22] registered [ 0.335117] acpiphp: Slot [23] registered [ 0.337113] acpiphp: Slot [24] registered [ 0.338175] acpiphp: Slot [25] registered [ 0.340108] acpiphp: Slot [26] registered [ 0.342118] acpiphp: Slot [27] registered [ 0.344103] acpiphp: Slot [28] registered [ 0.345112] acpiphp: Slot [29] registered [ 0.347108] acpiphp: Slot [30] registered [ 0.349142] acpiphp: Slot [31] registered [ 0.350113] PCI host bridge to bus 0000:00 [ 0.352022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.355024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.358027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.362033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.365031] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.368032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.370243] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.373832] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.377169] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.387709] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.393504] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.396026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.400023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.403021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.406430] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.409705] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.413040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.416795] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.422013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.440016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.446016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.451724] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.458018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.464020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.480021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.500153] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.514019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.525018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.553013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.567265] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.575014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.584016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.611018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.623073] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.635015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.645016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.667018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.682253] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.691016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.699020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.718025] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.730201] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.744019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.762021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.794020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.808693] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.812567] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.816555] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.819451] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.822244] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.828037] iommu: Default domain type: Passthrough [ 0.830527] SCSI subsystem initialized [ 0.832173] ACPI: bus type USB registered [ 0.834113] usbcore: registered new interface driver usbfs [ 0.836080] usbcore: registered new interface driver hub [ 0.839117] usbcore: registered new device driver usb [ 0.841190] pps_core: LinuxPPS API ver. 1 registered [ 0.843011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.846108] PTP clock support registered [ 0.849113] EDAC MC: Ver: 3.0.0 [ 0.851121] PCI: Using ACPI for IRQ routing [ 0.852891] NetLabel: Initializing [ 0.854013] NetLabel: domain hash size = 128 [ 0.856012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.858086] NetLabel: unlabeled traffic allowed by default [ 0.861295] vgaarb: loaded [ 0.862332] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.864017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.871853] clocksource: Switched to clocksource kvm-clock [ 0.985747] VFS: Disk quotas dquot_6.6.0 [ 0.987471] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.990392] *** VALIDATE ramfs *** [ 0.991967] *** VALIDATE hugetlbfs *** [ 0.994442] pnp: PnP ACPI init [ 0.997147] pnp: PnP ACPI: found 6 devices [ 1.019034] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.026457] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.029277] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.032121] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.035317] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.038547] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.041326] NET: Registered protocol family 2 [ 1.044077] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.049458] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.053801] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.062939] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.070426] TCP: Hash tables configured (established 65536 bind 65536) [ 1.076075] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.082339] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.088951] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.092353] NET: Registered protocol family 1 [ 1.095274] RPC: Registered named UNIX socket transport module. [ 1.098700] RPC: Registered udp transport module. [ 1.100971] RPC: Registered tcp transport module. [ 1.103196] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.106190] NET: Registered protocol family 44 [ 1.108216] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.110912] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.113448] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.116418] PCI: CLS 0 bytes, default 64 [ 1.119248] Unpacking initramfs... [ 2.548207] debug: unmapping init [mem 0xffff8ac23cc54000-0xffff8ac23ffbffff] [ 2.552722] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.555486] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.558800] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.064908] Initialise system trusted keyrings [ 3.067582] Key type blacklist registered [ 3.070771] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.083464] zbud: loaded [ 3.088315] *** VALIDATE nfs *** [ 3.090204] *** VALIDATE nfs4 *** [ 3.093139] pstore: using deflate compression [ 3.097111] Platform Keyring initialized [ 3.217617] NET: Registered protocol family 38 [ 3.219876] Key type asymmetric registered [ 3.222118] Asymmetric key parser 'x509' registered [ 3.224175] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.227737] io scheduler mq-deadline registered [ 3.229410] io scheduler kyber registered [ 3.231328] io scheduler bfq registered [ 3.233432] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.236884] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.239698] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.242819] ACPI: Power Button [PWRF] [ 3.248587] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.256317] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.275995] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.285856] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.304964] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.334814] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.361694] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.366941] Non-volatile memory driver v1.3 [ 3.369255] Linux agpgart interface v0.103 [ 3.406234] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.409841] vda: detected capacity change from 0 to 76677120 [ 3.441612] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.445152] vdb: detected capacity change from 0 to 1073741824 [ 3.460273] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.464987] vdc: detected capacity change from 0 to 2621440000 [ 3.483672] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.486392] vdd: detected capacity change from 0 to 2621440000 [ 3.503944] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.507301] vde: detected capacity change from 0 to 4294967296 [ 3.522490] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.527394] vdf: detected capacity change from 0 to 4294967296 [ 3.538241] libphy: Fixed MDIO Bus: probed [ 3.543419] usbcore: registered new interface driver usbserial_generic [ 3.546103] usbserial: USB Serial support registered for generic [ 3.548818] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.553511] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.555553] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.558923] mousedev: PS/2 mouse device common for all mice [ 3.563456] rtc_cmos 00:05: RTC can wake from S4 [ 3.567315] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.567528] rtc_cmos 00:05: registered as rtc0 [ 3.573314] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.574059] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.580335] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.581216] intel_pstate: CPU model not supported [ 3.587551] hid: raw HID events driver (C) Jiri Kosina [ 3.590224] usbcore: registered new interface driver usbhid [ 3.592377] usbhid: USB HID core driver [ 3.594273] drop_monitor: Initializing network drop monitor service [ 3.597186] Initializing XFRM netlink socket [ 3.601509] NET: Registered protocol family 10 [ 3.605662] Segment Routing with IPv6 [ 3.607354] NET: Registered protocol family 17 [ 3.610129] mpls_gso: MPLS GSO support [ 3.615510] RAS: Correctable Errors collector initialized. [ 3.617357] AVX version of gcm_enc/dec engaged. [ 3.618572] AES CTR mode by8 optimization enabled [ 3.694257] sched_clock: Marking stable (3694227140, 0)->(4630198773, -935971633) [ 3.700253] registered taskstats version 1 [ 3.703354] Loading compiled-in X.509 certificates [ 3.706507] zswap: loaded using pool lzo/zbud [ 3.732991] Key type big_key registered [ 3.745561] Key type encrypted registered [ 3.747798] ima: No TPM chip found, activating TPM-bypass! [ 3.750346] ima: Allocated hash algorithm: sha1 [ 3.752413] ima: No architecture policies found [ 3.754383] evm: Initialising EVM extended attributes: [ 3.756653] evm: security.selinux [ 3.758123] evm: security.ima [ 3.759589] evm: security.capability [ 3.761187] evm: HMAC attrs: 0x1 [ 3.763486] rtc_cmos 00:05: setting system clock to 2026-09-03 12:41:56 UTC (1788439316) [ 3.771830] debug: unmapping init [mem 0xffffffffaac03000-0xffffffffaadfffff] [ 3.775569] debug: unmapping init [mem 0xffffffffa9982000-0xffffffffa9c58fff] [ 3.786778] Write protecting the kernel read-only data: 28672k [ 3.790178] debug: unmapping init [mem 0xffffffffa8003000-0xffffffffa81fffff] [ 3.793355] debug: unmapping init [mem 0xffffffffa8914000-0xffffffffa89fffff] [ 3.826847] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.836650] systemd[1]: Detected virtualization kvm. [ 3.838890] systemd[1]: Detected architecture x86-64. [ 3.841148] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.870496] systemd[1]: No hostname configured. [ 3.872452] systemd[1]: Set hostname to . [ 3.874845] random: systemd: uninitialized urandom read (16 bytes read) [ 3.877590] systemd[1]: Initializing machine ID from random generator. [ 3.922788] random: ln: uninitialized urandom read (6 bytes read) [ 4.002401] random: systemd: uninitialized urandom read (16 bytes read) [ 4.006264] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.011434] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.017562] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.644095] device-mapper: uevent: version 1.0.3 [ 4.646605] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices...[ 5.043612] random: fast init done Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.400904] virtio_net virtio0 ens2: renamed from eth0 [ 5.449379] scsi host0: ata_piix [ 5.500246] scsi host1: ata_piix [ 5.503916] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.506700] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.933093] dracut-initqueue[591]: RTNETLINK answers: File exists [ 10.267578] random: crng init done [ 10.269630] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.775858] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.030422] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.309883] SELinux: Disabled at runtime. [ 12.374627] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.387173] systemd[1]: Detected virtualization kvm. [ 12.389558] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.933588] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.937668] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.943771] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.956646] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.960947] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.970332] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.976980] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Reached target Local Encrypted Volumes. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK [[ 13.128823] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS 0m] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 13.470972] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.804346] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.810990] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.003269] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.013949] EDAC sbridge: Ver: 1.1.2 [ 15.863297] Key type dns_resolver registered [ 16.185812] NFS: Registering the id_resolver key type [ 16.188520] Key type id_resolver registered [ 16.190986] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg334-server login: [ 74.714059] libcfs: loading out-of-tree module taints kernel. [ 74.820912] Key type ._llcrypt registered [ 74.823344] Key type .llcrypt registered [ 74.958559] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 94.075242] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 95.481625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 95.492552] alg: No test for adler32 (adler32-zlib) [ 96.956614] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 97.904042] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 99.639173] Key type lgssc registered [ 101.509630] Lustre: Echo OBD driver; http://www.lustre.org/ [ 120.949423] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 132.645870] hrtimer: interrupt took 7861588 ns [ 162.760209] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 177.192156] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 177.227811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 178.516854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 178.552123] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 178.611655] Lustre: lustre-MDT0000: new disk, initializing [ 178.713711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 178.742830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 182.850857] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 197.903610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 198.003382] Lustre: 6517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 198.040750] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 198.044778] Lustre: Skipped 1 previous similar message [ 198.229553] Lustre: lustre-MDT0001: new disk, initializing [ 198.333724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 198.349812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 198.357547] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 203.365747] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 208.685541] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 220.647575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 220.961975] Lustre: lustre-OST0000: new disk, initializing [ 220.976687] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 220.983315] Lustre: 8458:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 221.060029] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 223.542118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 223.565211] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 223.675821] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 229.651529] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 242.846085] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 243.052191] Lustre: lustre-OST0001: new disk, initializing [ 243.054905] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 243.059888] Lustre: 9531:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 243.116162] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 248.763835] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 249.389538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 249.400798] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 249.468282] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 260.899323] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 271.429989] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 278.444124] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing check_logdir /tmp/testlogs/ [ 285.281413] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing yml_node [ 290.496481] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 293.367410] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 295.970340] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 297.735391] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 08:46:47 EDT 2026 [ 316.326453] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 318.077979] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 336.353248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 336.379307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 336.392076] Lustre: Skipped 2 previous similar messages [ 342.347578] Lustre: server umount lustre-MDT0000 complete [ 346.079598] LustreError: 6510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439658 with bad export cookie 14213291588332820347 [ 346.084743] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 346.090191] LustreError: 6510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 346.591799] LustreError: 6526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 346.596968] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 346.607630] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 346.612349] LustreError: 6526:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 346.650510] Lustre: Skipped 4 previous similar messages [ 351.712488] LustreError: 10105:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 351.713987] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 351.738581] LustreError: 10105:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 351.764613] Lustre: Skipped 2 previous similar messages [ 352.594948] Lustre: server umount lustre-MDT0001 complete [ 356.653903] Lustre: server umount lustre-OST0000 complete [ 360.762738] Lustre: server umount lustre-OST0001 complete [ 368.164275] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 377.266994] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 413.382302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 413.638337] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 413.687355] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 413.790727] Lustre: lustre-MDT0000: new disk, initializing [ 413.936305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 413.956263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 418.757453] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 430.613512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 430.988860] Lustre: 15084:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 431.203841] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 431.215885] Lustre: Skipped 1 previous similar message [ 431.334242] Lustre: lustre-MDT0001: new disk, initializing [ 431.427994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 431.472225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 431.476237] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 438.238565] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 448.113723] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 453.653671] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 460.867510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 461.195446] Lustre: lustre-OST0000: new disk, initializing [ 461.213178] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 461.229158] Lustre: 17054:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 461.338367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 462.253647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 462.285157] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 462.404028] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 467.572118] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 476.691484] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 481.929356] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 482.297733] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 487.516604] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 487.888701] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 493.031062] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 493.039586] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 493.064362] LustreError: Skipped 1 previous similar message [ 493.072344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 493.090756] Lustre: Skipped 1 previous similar message [ 496.919208] Lustre: server umount lustre-OST0000 complete [ 503.264978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 503.272461] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 503.279165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 503.284361] Lustre: Skipped 2 previous similar messages [ 503.307219] Lustre: Skipped 1 previous similar message [ 507.341815] Lustre: server umount lustre-MDT0000 complete [ 508.389923] LustreError: 15089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 508.426875] LustreError: 15089:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 511.557431] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439824 with bad export cookie 14213291588332821880 [ 511.563907] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 511.568847] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 512.310109] Lustre: server umount lustre-MDT0001 complete [ 516.659330] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 08:50:26 (1788439826) [ 518.365507] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 519.724490] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 08:50:30 (1788439830) [ 525.212371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 525.567400] LustreError: 19464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 525.662234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 530.908313] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 530.924695] LustreError: 19463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 540.587194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 545.897218] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 553.458372] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 559.599109] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 566.762161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 573.683399] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 580.785811] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 595.937363] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 595.941258] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 595.953260] LustreError: Skipped 1 previous similar message [ 595.988415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 597.596774] Lustre: server umount lustre-OST0000 complete [ 602.080607] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 606.178367] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 606.189411] Lustre: Skipped 2 previous similar messages [ 607.738769] Lustre: server umount lustre-MDT0000 complete [ 611.196286] LustreError: 19444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439923 with bad export cookie 14213291588332822769 [ 611.206804] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 611.208878] LustreError: 19444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 611.461616] Lustre: server umount lustre-MDT0001 complete [ 622.959615] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 625.429294] Key type lgssc unregistered [ 625.730765] LNet: 22996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 625.740089] LNetError: 22996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 625.752823] LNet: Removed LNI 192.168.203.134@tcp [ 626.586153] Key type .llcrypt unregistered [ 626.593943] Key type ._llcrypt unregistered [ 652.378024] Key type ._llcrypt registered [ 652.380563] Key type .llcrypt registered [ 652.530523] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 654.623091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 654.900450] alg: No test for adler32 (adler32-zlib) [ 656.344305] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 656.860085] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 658.559122] Key type lgssc registered [ 660.081419] Lustre: Echo OBD driver; http://www.lustre.org/ [ 669.059504] loop: module loaded [ 669.169224] loop0: detected capacity change from 0 to 204800000 [ 669.565979] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 669.606053] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 669.831788] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 676.799387] loop0: detected capacity change from 0 to 204800000 [ 676.902845] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 676.919052] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 676.931333] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 683.232683] loop0: detected capacity change from 0 to 204800000 [ 689.771554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 689.811319] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 691.120167] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 691.208388] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 691.287599] Lustre: lustre-MDT0000: new disk, initializing [ 691.376129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 691.412173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 696.760621] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 706.033476] loop1: detected capacity change from 0 to 204800000 [ 711.468665] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 711.704452] Lustre: lustre-OST07c6: new disk, initializing [ 711.708642] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 711.712259] Lustre: Skipped 1 previous similar message [ 711.716455] Lustre: 26179:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 711.782878] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 718.061041] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 720.068983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 720.091081] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 720.136544] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 722.740416] Lustre: 25167:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 722.749683] Lustre: 25167:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 722.754320] Lustre: 25167:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 722.761304] Lustre: 25167:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 722.772376] Lustre: 25167:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 722.777647] Lustre: 25167:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 739.505901] Lustre: 25167:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 739.513142] Lustre: 25167:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 739.518946] Lustre: 25167:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 739.525637] Lustre: 25167:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 739.531956] Lustre: 25167:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 739.540334] Lustre: 25167:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 744.448837] Lustre: 25166:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 744.461907] Lustre: 25166:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 744.476050] Lustre: 25166:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 744.495533] Lustre: 25166:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 744.510687] Lustre: 25166:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 744.522686] Lustre: 25166:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 744.535567] Lustre: 25166:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 744.546849] Lustre: 25166:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 744.555522] Lustre: 25166:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 744.563963] Lustre: 25166:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 744.577693] Lustre: 25166:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 744.588977] Lustre: 25166:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 747.789243] Lustre: 26187:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 747.809709] Lustre: 26187:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 747.827484] Lustre: 26187:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 747.843796] Lustre: 26187:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 747.852251] Lustre: 26187:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 747.863746] Lustre: 26187:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 747.878758] Lustre: 26187:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 747.889507] Lustre: 26187:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 747.897803] Lustre: 26187:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 747.907126] Lustre: 26187:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 747.914746] Lustre: 26187:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 747.921845] Lustre: 26187:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.912897] Lustre: 26187:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 752.922089] Lustre: 26187:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 752.934773] Lustre: 26187:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 752.941631] Lustre: 26187:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.951443] Lustre: 26187:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 752.958525] Lustre: 26187:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.966585] Lustre: 26187:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 752.972736] Lustre: 26187:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.977591] Lustre: 26187:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 752.982403] Lustre: 26187:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.987145] Lustre: 26187:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 752.991319] Lustre: 26187:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.851256] Lustre: 25165:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 763.859466] Lustre: 25165:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 763.863234] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 763.871557] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 763.877637] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 763.885564] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 763.894135] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 763.906587] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 763.914755] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 763.922233] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 763.928615] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 763.935376] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 812.515544] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 812.516261] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 812.564345] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 816.685531] Lustre: server umount lustre-OST07c6 complete [ 827.576400] Lustre: server umount lustre-MDT0000 complete [ 844.747460] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 848.129733] Key type lgssc unregistered [ 848.467617] LNet: 28736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 848.475248] LNetError: 28736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 848.492219] LNet: Removed LNI 192.168.203.134@tcp [ 849.397497] Key type .llcrypt unregistered [ 849.399353] Key type ._llcrypt unregistered [ 868.677225] Key type ._llcrypt registered [ 868.681334] Key type .llcrypt registered [ 868.833264] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 870.241517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 870.330401] alg: No test for adler32 (adler32-zlib) [ 871.350564] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 871.605670] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 873.303343] Key type lgssc registered [ 874.614101] Lustre: Echo OBD driver; http://www.lustre.org/ [ 882.960075] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 882.990490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 884.466708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 889.098148] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 894.944953] LustreError: 30004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 894.967986] LustreError: 30004:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 897.928878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 898.420837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 903.886376] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 912.126620] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 917.713678] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 924.589992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 925.084156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 930.248604] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 932.344646] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 940.152044] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 958.953702] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 958.955393] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 958.977672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 959.020139] Lustre: Skipped 1 previous similar message [ 963.579857] Lustre: server umount lustre-OST0000 complete [ 969.190292] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 969.211944] Lustre: Skipped 1 previous similar message [ 969.220123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 969.235468] Lustre: Skipped 2 previous similar messages [ 973.849491] Lustre: server umount lustre-MDT0000 complete [ 974.304612] LustreError: 30004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 974.327531] LustreError: 30004:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 977.793465] LustreError: 29986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440290 with bad export cookie 9091523292317324302 [ 977.797567] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 977.803298] LustreError: 29986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 978.309595] Lustre: server umount lustre-MDT0001 complete [ 990.339362] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 993.567464] Key type lgssc unregistered [ 993.902930] LNet: 33743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 993.911204] LNetError: 33743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 993.943879] LNet: Removed LNI 192.168.203.134@tcp [ 994.963413] Key type .llcrypt unregistered [ 994.964768] Key type ._llcrypt unregistered [ 1015.158028] Key type ._llcrypt registered [ 1015.163868] Key type .llcrypt registered [ 1015.363673] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1016.825267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1016.976032] alg: No test for adler32 (adler32-zlib) [ 1018.132284] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1018.601763] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 1020.370421] Key type lgssc registered [ 1021.894879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1029.836111] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1029.858288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1031.272343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1036.160482] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1040.872066] LustreError: 35008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1040.890617] LustreError: 35008:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1044.193156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1044.649280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1049.373185] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1056.537506] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1062.435764] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1069.376242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1069.845453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1075.951684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1076.354979] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1084.086045] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1100.773439] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1100.778946] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1100.790715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1100.809807] Lustre: Skipped 1 previous similar message [ 1103.399256] Lustre: server umount lustre-OST0000 complete [ 1111.008711] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1111.019699] Lustre: Skipped 1 previous similar message [ 1111.025947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1111.034222] Lustre: Skipped 1 previous similar message [ 1113.347545] Lustre: server umount lustre-MDT0000 complete [ 1116.142565] LustreError: 35738:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1116.177096] LustreError: 35738:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1117.321349] LustreError: 37811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440430 with bad export cookie 2666979324948130392 [ 1117.323749] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1117.331262] LustreError: 37811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1117.759705] Lustre: server umount lustre-MDT0001 complete [ 1128.774039] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 1131.753328] Key type lgssc unregistered [ 1132.080371] LNet: 38563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1132.086639] LNetError: 38563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1133.158787] LNet: Removed LNI 192.168.203.134@tcp [ 1134.120435] Key type .llcrypt unregistered [ 1134.123476] Key type ._llcrypt unregistered [ 1159.042125] Key type ._llcrypt registered [ 1159.043936] Key type .llcrypt registered [ 1159.223123] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1160.413126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1160.449291] alg: No test for adler32 (adler32-zlib) [ 1161.581488] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1161.806197] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 1163.495201] Key type lgssc registered [ 1164.839672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1173.192458] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1173.212223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1174.727949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1179.570192] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1185.248905] LustreError: 39991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1185.263105] LustreError: 39991:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1187.715610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1188.172835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1192.494232] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1200.992579] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1207.357437] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1216.084115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1216.544481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1219.652725] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1223.501973] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1231.556462] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1239.338523] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.34@tcp (stopping) [ 1239.520601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1239.533513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1240.503790] Lustre: server umount lustre-MDT0000 complete [ 1243.105859] LustreError: 41025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1243.115416] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1243.147245] Lustre: Skipped 1 previous similar message [ 1243.933406] LustreError: 39970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440556 with bad export cookie 14321135409590613204 [ 1243.941708] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1243.951922] LustreError: 39970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1244.436548] LustreError: 39985:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1244.472159] LustreError: 39985:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1244.481725] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.34@tcp (stopping) [ 1244.492724] Lustre: Skipped 1 previous similar message [ 1248.224982] LustreError: 41831:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1248.227547] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1248.262989] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1250.473634] Lustre: server umount lustre-MDT0001 complete [ 1271.874621] Lustre: server umount lustre-OST0000 complete [ 1287.309104] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 1290.144747] Key type lgssc unregistered [ 1290.462204] LNet: 43850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1290.475284] LNetError: 43850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1290.485863] LNet: Removed LNI 192.168.203.134@tcp [ 1291.204436] Key type .llcrypt unregistered [ 1291.208748] Key type ._llcrypt unregistered [ 1315.145880] Key type ._llcrypt registered [ 1315.150548] Key type .llcrypt registered [ 1315.287553] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1316.380588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1316.392220] alg: No test for adler32 (adler32-zlib) [ 1317.700454] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1318.075426] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 1319.799180] Key type lgssc registered [ 1321.285272] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1329.699237] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1329.734286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1331.365066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1336.598270] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1341.924838] LustreError: 45280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1341.955781] LustreError: 45280:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1345.585199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1346.092869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1351.518759] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1360.104674] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1366.325739] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1373.497529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1373.809286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1377.918534] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1380.465638] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1388.963824] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1398.753905] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1398.774846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1398.796985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1401.127464] Lustre: lustre-OST0000: Not available for connect from 192.168.203.34@tcp (stopping) [ 1402.853709] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1402.862195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1402.871071] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1402.919508] Lustre: Skipped 1 previous similar message [ 1404.282113] Lustre: server umount lustre-OST0000 complete [ 1422.499408] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1422.506261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1422.525783] Lustre: Skipped 1 previous similar message [ 1422.547536] Lustre: Skipped 1 previous similar message [ 1427.936718] LustreError: 45280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1428.041827] Lustre: server umount lustre-MDT0000 complete [ 1432.389658] LustreError: 45261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440745 with bad export cookie 11711894801512610322 [ 1432.390603] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1432.403859] LustreError: 45261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1439.251431] Lustre: server umount lustre-MDT0001 complete [ 1450.774386] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 1452.981323] Key type lgssc unregistered [ 1453.362680] LNet: 48987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1453.371723] LNetError: 48987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1453.383791] LNet: Removed LNI 192.168.203.134@tcp [ 1454.069048] Key type .llcrypt unregistered [ 1454.071062] Key type ._llcrypt unregistered [ 1480.100743] Key type ._llcrypt registered [ 1480.105745] Key type .llcrypt registered [ 1480.228150] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1481.465218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1481.518434] alg: No test for adler32 (adler32-zlib) [ 1482.589462] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1482.897554] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 1484.599166] Key type lgssc registered [ 1485.901656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1493.990401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1494.007079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1495.309535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1499.283767] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1504.739705] LustreError: 50401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1504.783499] LustreError: 50401:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1508.080652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1508.524598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1514.338302] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1524.324878] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1529.293555] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1535.849753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1536.394695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1539.458759] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1542.229843] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1549.861571] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1558.736173] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1562.159330] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1574.248667] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-03 9h08m04s [ 1575.677980] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1579.492600] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1579.500635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1579.529246] Lustre: Skipped 1 previous similar message [ 1580.579701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1580.582438] Lustre: Skipped 2 previous similar messages [ 1582.370573] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.34@tcp (stopping) [ 1583.967493] Lustre: server umount lustre-MDT0000 complete [ 1586.658979] LustreError: 50401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1586.680684] LustreError: 50401:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1587.204230] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1589.514315] LustreError: 50396:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1592.810899] LustreError: 51128:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1592.830068] LustreError: 51128:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1592.893793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1592.993383] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1593.177886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1593.251886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1594.159294] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1597.307700] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1600.800974] Lustre: DEBUG MARKER: Wait for df (36737) ... [ 1602.254254] Lustre: DEBUG MARKER: done [ 1608.673324] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1608.678320] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1608.686023] Lustre: Skipped 1 previous similar message [ 1608.699415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1608.702071] Lustre: Skipped 1 previous similar message [ 1609.702860] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1612.944697] Lustre: server umount lustre-OST0000 complete [ 1619.937661] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1619.940602] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1619.940708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1619.940712] Lustre: Skipped 2 previous similar messages [ 1619.957196] Lustre: Skipped 1 previous similar message [ 1623.569711] Lustre: server umount lustre-MDT0000 complete [ 1626.966530] LustreError: 53027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440939 with bad export cookie 10409566342534890525 [ 1626.975936] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1626.996908] LustreError: 53027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1627.309787] Lustre: server umount lustre-MDT0001 complete [ 1638.359650] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 1641.814778] Key type lgssc unregistered [ 1642.281798] LNet: 55324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1642.290306] LNetError: 55324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1642.341817] LNet: Removed LNI 192.168.203.134@tcp [ 1643.444815] Key type .llcrypt unregistered [ 1643.446552] Key type ._llcrypt unregistered [ 1684.113569] Key type ._llcrypt registered [ 1684.119308] Key type .llcrypt registered [ 1684.217204] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1685.272574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1685.361271] alg: No test for adler32 (adler32-zlib) [ 1686.691276] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1687.030026] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 1688.735179] Key type lgssc registered [ 1689.873158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1696.989499] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1697.013896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1698.303257] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1698.334636] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1698.476089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1703.020739] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1711.408913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1711.500116] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1711.530528] Lustre: 57530:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1711.800523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1716.793645] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1724.248887] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1729.900185] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1736.239587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1736.478072] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1736.658150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1742.638819] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1746.541748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1749.304721] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1756.102334] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-03 9h11m06s [ 1757.564442] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1764.254606] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1766.540537] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1766.583460] LustreError: 57537:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1786.665730] Lustre: lustre-MDT0000: Client 975e150d-6b11-461d-aaaa-b19c8f98f3ee (at 192.168.203.34@tcp) reconnecting [ 1806.591128] LustreError: 57537:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1808.965366] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1812.294896] Lustre: DEBUG MARKER: done [ 1821.668222] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1821.674928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1821.678706] Lustre: Skipped 1 previous similar message [ 1826.395762] Lustre: server umount lustre-OST0000 complete [ 1831.909973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1831.922174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1831.946762] Lustre: Skipped 2 previous similar messages [ 1831.967635] Lustre: Skipped 1 previous similar message [ 1832.995617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1833.000547] Lustre: Skipped 1 previous similar message [ 1837.082931] Lustre: server umount lustre-MDT0000 complete [ 1840.711856] LustreError: 57522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441153 with bad export cookie 2055024470336285171 [ 1840.723390] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1840.730938] LustreError: 57522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1841.059151] Lustre: server umount lustre-MDT0001 complete [ 1852.943716] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 1855.724784] Key type lgssc unregistered [ 1855.964471] LNet: 61867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1855.976863] LNetError: 61867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1855.996498] LNet: Removed LNI 192.168.203.134@tcp [ 1856.828314] Key type .llcrypt unregistered [ 1856.837000] Key type ._llcrypt unregistered [ 1904.014245] Key type ._llcrypt registered [ 1904.016530] Key type .llcrypt registered [ 1904.131228] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1905.600933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1905.636178] alg: No test for adler32 (adler32-zlib) [ 1906.845497] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1907.222438] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 1908.967168] Key type lgssc registered [ 1910.134328] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1912.929822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1912.955029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1914.308844] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1914.374072] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1914.541826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1915.620128] Lustre: Failing over lustre-MDT0000 [ 1915.937281] Lustre: server umount lustre-MDT0000 complete [ 1924.327370] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:13:54 (1788441234) [ 1933.248681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1933.999361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1938.650893] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1947.125938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1947.222043] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1947.254943] Lustre: 64666:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1947.569359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1952.508833] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1959.762188] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1965.099927] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1971.722411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1971.924347] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1972.158438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1978.182710] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1980.577231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1985.161510] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1991.147632] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1997.060928] Lustre: server umount lustre-MDT0000 complete [ 1998.825707] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1998.837985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2000.734704] LustreError: 64658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441313 with bad export cookie 14904152713336919977 [ 2000.735638] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2000.758909] LustreError: 64658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2001.020377] Lustre: server umount lustre-MDT0001 complete [ 2004.245573] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2008.706723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2017.040373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2018.271245] Lustre: 63588:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441314/real 1788441314] req@000000002af65141 x1875316539692032/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788441330 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2018.296145] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2021.407372] Lustre: 63586:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441318/real 1788441318] req@0000000097fa83cc x1875316539692416/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788441334 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2021.443344] Lustre: 63586:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2025.919691] LustreError: 66513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2025.937594] LustreError: 66513:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2026.051701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2026.176187] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2027.039111] Lustre: 63586:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441323/real 1788441323] req@00000000fed6ab9a x1875316539692928/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788441339 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2027.091248] Lustre: 63586:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2029.090944] LustreError: 68435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2029.105260] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2029.116309] LustreError: 68435:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2030.362432] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2032.099420] Lustre: 63587:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441328/real 1788441328] req@000000007d8ef8e2 x1875316539693184/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788441344 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2032.136270] Lustre: 63587:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2034.145533] LustreError: 68434:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2034.174959] LustreError: 68434:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2038.544747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2038.886266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2043.472901] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2044.419706] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2051.070913] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2056.684851] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2063.324482] Lustre: server umount lustre-MDT0000 complete [ 2064.356857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2064.366691] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2064.381583] Lustre: Skipped 1 previous similar message [ 2064.385622] LustreError: 66514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2066.678371] LustreError: 66529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441379 with bad export cookie 14904152713336922252 [ 2066.682414] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2066.700051] LustreError: 66529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2069.084173] Lustre: server umount lustre-MDT0001 complete [ 2074.774794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2083.604682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2085.343152] Lustre: 63588:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441382/real 1788441382] req@000000007c2af764 x1875316539718144/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788441398 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2085.363410] Lustre: 63588:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2085.367809] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2085.375017] Lustre: Skipped 2 previous similar messages [ 2091.977384] LustreError: 66514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2092.000125] LustreError: 66514:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2092.063615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2092.116564] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2096.683727] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2099.168502] Lustre: 63588:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441395/real 1788441395] req@000000005636c17a x1875316539718784/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788441411 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2099.222444] Lustre: 63588:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2101.217401] LustreError: 71120:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2101.227875] LustreError: 71120:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2104.321843] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2104.444544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2109.830803] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2110.441597] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2118.537313] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2124.551373] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2136.036456] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2136.037023] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2136.049293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2136.100174] Lustre: Skipped 2 previous similar messages [ 2137.785100] Lustre: server umount lustre-MDT0000 complete [ 2141.165976] LustreError: 71119:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2141.195873] LustreError: 71119:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2141.704871] LustreError: 64658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441454 with bad export cookie 14904152713336923631 [ 2141.713293] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2141.716182] LustreError: 64658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2142.081548] Lustre: server umount lustre-MDT0001 complete [ 2148.269450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2151.443842] Lustre: DEBUG MARKER: files compared the same [ 2162.435528] Lustre: server umount lustre-OST0000 complete [ 2178.047965] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 2181.006435] Key type lgssc unregistered [ 2181.272971] LNet: 74393:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2181.285510] LNetError: 74393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2181.314791] LNet: Removed LNI 192.168.203.134@tcp [ 2182.405702] Key type .llcrypt unregistered [ 2182.411581] Key type ._llcrypt unregistered [ 2206.174974] Key type ._llcrypt registered [ 2206.176721] Key type .llcrypt registered [ 2206.306448] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 2207.340846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2207.455265] alg: No test for adler32 (adler32-zlib) [ 2208.453786] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2208.734620] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 2210.408537] Key type lgssc registered [ 2211.445959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2219.775910] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2219.788306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2221.256386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2225.756813] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug +malloc all [ 2230.756078] LustreError: 75807:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2230.778368] LustreError: 75807:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2234.418661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2234.756523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2239.361373] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug +malloc all [ 2247.115726] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2252.740836] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2259.084669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2259.394831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2265.483817] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug +malloc all [ 2265.589246] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2272.608329] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2280.928478] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2280.934170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2280.934855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2280.950496] Lustre: Skipped 1 previous similar message [ 2280.968788] Lustre: Skipped 1 previous similar message [ 2286.050089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2286.063158] Lustre: Skipped 1 previous similar message [ 2286.872802] Lustre: server umount lustre-OST0000 complete [ 2291.168351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2291.204950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2291.214810] Lustre: Skipped 1 previous similar message [ 2296.296641] LustreError: 76536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2296.315470] LustreError: 76536:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2296.455411] Lustre: server umount lustre-MDT0000 complete [ 2300.959585] LustreError: 75788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441613 with bad export cookie 10477504098631192098 [ 2300.963414] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2300.976220] LustreError: 75788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2301.342775] Lustre: server umount lustre-MDT0001 complete [ 2312.784989] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 2315.353944] Key type lgssc unregistered [ 2315.601713] LNet: 79340:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2315.611922] LNetError: 79340:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2315.644358] LNet: Removed LNI 192.168.203.134@tcp [ 2316.557129] Key type .llcrypt unregistered [ 2316.567628] Key type ._llcrypt unregistered [ 2345.560900] Key type ._llcrypt registered [ 2345.563670] Key type .llcrypt registered [ 2345.679530] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 2346.981130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2347.145064] alg: No test for adler32 (adler32-zlib) [ 2348.509381] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2348.830849] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 2350.599529] Key type lgssc registered [ 2351.698708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2358.158350] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2358.183266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2373.728366] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2378.847494] LustreError: 80728:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.134@tcp: failed processing log, type 4: rc = -110 [ 2405.535178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2412.043667] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2420.509108] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2428.654592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2428.783809] Lustre: *** cfs_fail_loc=706, val=0*** [ 2428.796418] LustreError: 80063:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d530ff48 x1875317003127040/t0(0) o250->MGC192.168.203.134@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2429.202871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2429.284492] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2433.771915] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2434.530226] LustreError: 80754:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2434.570252] LustreError: 80754:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2439.650615] LustreError: 80753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2440.682025] LustreError: 80754:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2440.734688] LustreError: 80754:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2442.377113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2442.921234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2447.876812] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2456.213947] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2461.999069] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2465.258411] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2465.266395] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2465.283704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2468.322982] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2468.335309] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2468.347055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2470.565540] Lustre: server umount lustre-OST0000 complete [ 2478.560234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2478.571080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2478.591921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2478.594906] Lustre: Skipped 1 previous similar message [ 2482.617456] Lustre: server umount lustre-MDT0000 complete [ 2484.208631] LustreError: 82457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2484.230239] LustreError: 82457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2486.785893] LustreError: 80736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441799 with bad export cookie 13650831449559452003 [ 2486.806337] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2487.128965] Lustre: server umount lustre-MDT0001 complete [ 2498.555292] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 2501.134538] Key type lgssc unregistered [ 2501.441697] LNet: 84341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2501.450675] LNetError: 84341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2501.465356] LNet: Removed LNI 192.168.203.134@tcp [ 2502.554375] Key type .llcrypt unregistered [ 2502.559854] Key type ._llcrypt unregistered [ 2529.977135] Key type ._llcrypt registered [ 2529.978632] Key type .llcrypt registered [ 2530.053144] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 2530.654370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2530.726145] alg: No test for adler32 (adler32-zlib) [ 2531.784634] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2532.260118] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 2534.111209] Key type lgssc registered [ 2535.426690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2545.240803] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2545.272092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2550.913314] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2558.854260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2559.564469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2563.978270] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2564.586578] LustreError: 86445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2564.596262] LustreError: 86445:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2569.697712] LustreError: 86449:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2569.732017] LustreError: 86449:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2571.912612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2572.182368] LustreError: 86445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2572.203052] LustreError: 86445:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2572.215933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2577.385565] LustreError: 86449:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2577.411945] LustreError: 86449:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2578.905773] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2582.519173] LustreError: 86444:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2582.540731] LustreError: 86444:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2588.680287] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2588.698410] Lustre: MGS: Client 24826e09-d60c-4839-b606-a001e13955e2 (at 0@lo) reconnecting [ 2588.743416] LustreError: 86459:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2589.061859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2589.191480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2594.127485] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2609.637237] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2609.661721] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2609.662120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2609.678349] Lustre: Skipped 1 previous similar message [ 2613.813790] Lustre: server umount lustre-OST0000 complete [ 2619.873812] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2619.874810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2619.895563] Lustre: Skipped 2 previous similar messages [ 2619.906320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2623.964882] Lustre: server umount lustre-MDT0000 complete [ 2625.008516] LustreError: 86448:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2625.033098] LustreError: 86448:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2627.892442] Lustre: server umount lustre-MDT0001 complete [ 2631.416831] Lustre: server umount MGS complete [ 2643.098802] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:25:53 (1788441953) [ 2671.851607] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 2682.181082] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 2718.416210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2723.491629] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2733.815657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2734.084862] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2734.179401] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2734.338762] Lustre: lustre-MDT0001: new disk, initializing [ 2734.436805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2734.473844] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2734.476949] Lustre: 93616:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2734.484555] LustreError: 93616:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2739.602733] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2750.364443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2750.628855] Lustre: lustre-OST0000: new disk, initializing [ 2750.633812] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2750.644872] Lustre: 94383:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2750.744943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2757.728469] Lustre: 94674:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000054ac0663 x1875317195055488/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788442086 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2758.126098] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2768.508807] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2775.599699] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2775.612952] Lustre: MGS: Client 2cf85653-62b0-4eeb-ad19-637e20442c1b (at 0@lo) reconnecting [ 2775.659480] Lustre: 95428:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2775.688631] LustreError: 93602:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2775.707136] LustreError: 93602:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2775.725420] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2775.863886] Lustre: lustre-MDT0000: new disk, initializing [ 2776.106409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2776.145170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2776.192677] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2777.250378] Lustre: 95452:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000b50d5c1d x1875317195063040/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788442105 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2781.174546] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2782.375586] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2785.266972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2785.274305] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2786.431678] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2812.384545] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2812.390438] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2812.397496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2812.409728] Lustre: Skipped 1 previous similar message [ 2814.944405] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2814.958747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2814.978257] Lustre: Skipped 1 previous similar message [ 2820.064816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2820.081842] Lustre: Skipped 1 previous similar message [ 2825.187301] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2825.376880] Lustre: server umount lustre-OST0000 complete [ 2830.306316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2830.310200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2830.312084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2830.328587] Lustre: Skipped 1 previous similar message [ 2830.340756] Lustre: Skipped 2 previous similar messages [ 2835.574510] Lustre: server umount lustre-MDT0000 complete [ 2845.464871] Lustre: server umount lustre-MDT0001 complete [ 2849.324359] Lustre: server umount MGS complete [ 2860.315486] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:29:30 (1788442170) [ 2878.881988] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 2882.129293] Key type lgssc unregistered [ 2882.479786] LNet: 97951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2882.492712] LNetError: 97951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2882.515820] LNet: Removed LNI 192.168.203.134@tcp [ 2883.661736] Key type .llcrypt unregistered [ 2883.667259] Key type ._llcrypt unregistered [ 2902.562544] Key type ._llcrypt registered [ 2902.564966] Key type .llcrypt registered [ 2902.685239] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 2903.689153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2903.737416] alg: No test for adler32 (adler32-zlib) [ 2904.855820] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2905.113101] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 2906.887221] Key type lgssc registered [ 2908.193937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2916.089149] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2916.114027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2917.630941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2922.201452] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2928.102724] LustreError: 99171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2928.125908] LustreError: 99171:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2931.532328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2931.942764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2937.139684] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2945.235353] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2951.283262] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2958.555075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2958.968731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2963.072140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2966.074310] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2974.393100] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2986.979422] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2986.981479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2987.007836] Lustre: Skipped 1 previous similar message [ 2987.010971] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2987.039965] Lustre: Skipped 1 previous similar message [ 2991.443927] Lustre: server umount lustre-OST0000 complete [ 2997.217615] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2997.227258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2997.239394] Lustre: Skipped 1 previous similar message [ 2997.263207] Lustre: Skipped 1 previous similar message [ 3001.639738] Lustre: server umount lustre-MDT0000 complete [ 3002.336356] LustreError: 99166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3002.353211] LustreError: 99166:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3004.940794] LustreError: 99150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442317 with bad export cookie 1338008972575863402 [ 3004.946134] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3004.950866] LustreError: 99150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3005.163257] Lustre: server umount lustre-MDT0001 complete [ 3017.184806] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 3020.721264] Key type lgssc unregistered [ 3021.244558] LNet: 102706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3021.254813] LNetError: 102706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3021.285677] LNet: Removed LNI 192.168.203.134@tcp [ 3022.349464] Key type .llcrypt unregistered [ 3022.352342] Key type ._llcrypt unregistered [ 3036.902782] Key type ._llcrypt registered [ 3036.908197] Key type .llcrypt registered [ 3037.084943] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3037.870353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3037.882224] alg: No test for adler32 (adler32-zlib) [ 3038.837612] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3039.028563] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 3040.680879] Key type lgssc registered [ 3041.828944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3048.026050] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3048.040713] LustreError: 103780:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3048.085761] LustreError: 103787:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3048.095494] LustreError: 103780:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4956 [ 3048.101910] LustreError: 103780:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3048.112471] LustreError: 103780:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3048.115872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3049.686023] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3055.071772] LustreError: 103824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3055.090742] LustreError: 103824:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3057.055435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3057.276920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3061.746714] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3067.448334] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing lsmod [ 3070.551048] LustreError: 104986:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3070.599696] LustreError: 104993:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3070.608948] LustreError: 104986:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4962 [ 3070.621460] LustreError: 104986:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3070.627815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3070.636598] LustreError: 104986:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3071.177159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3076.577019] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3076.583604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3076.595584] Lustre: Skipped 2 previous similar messages [ 3076.596150] Lustre: Skipped 1 previous similar message [ 3081.703298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3081.711982] Lustre: Skipped 3 previous similar messages [ 3082.721495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3082.728156] Lustre: Skipped 2 previous similar messages [ 3087.845610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3087.854055] Lustre: Skipped 2 previous similar messages [ 3088.351142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3088.610718] Lustre: server umount lustre-MDT0000 complete [ 3092.204254] LustreError: 103802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442404 with bad export cookie 17910562254721927710 [ 3092.215350] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3092.217970] LustreError: 103802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3092.961350] LustreError: 105181:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3092.967443] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3093.003153] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3093.034038] LustreError: 104255:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000008fbc9e41 x1875317726663680/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3093.219903] Lustre: server umount lustre-MDT0001 complete [ 3098.865278] Lustre: server umount lustre-OST0000 complete [ 3107.399529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3107.945936] LustreError: 106058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3108.003688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3113.110669] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3113.445705] LustreError: 106059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3117.554349] LustreError: 106058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3121.828744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3122.269459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3127.175929] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3135.091549] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3141.475372] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3149.051915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3149.366680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3152.466387] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3156.482674] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3164.937017] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3177.952587] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3177.971913] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3177.998409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3182.058372] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3182.071570] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3184.030853] Lustre: server umount lustre-OST0000 complete [ 3188.705352] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3188.715598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3193.639728] Lustre: server umount lustre-MDT0000 complete [ 3197.412703] LustreError: 106058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3197.445077] LustreError: 106058:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3197.496281] LustreError: 106040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442510 with bad export cookie 17910562254721928396 [ 3197.512330] LustreError: 106040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3197.514909] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3197.947778] Lustre: server umount lustre-MDT0001 complete [ 3209.197752] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 3212.444885] Key type lgssc unregistered [ 3212.787846] LNet: 109614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3212.798682] LNetError: 109614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3212.822407] LNet: Removed LNI 192.168.203.134@tcp [ 3213.779695] Key type .llcrypt unregistered [ 3213.781571] Key type ._llcrypt unregistered [ 3240.491446] Key type ._llcrypt registered [ 3240.493393] Key type .llcrypt registered [ 3240.592906] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3242.103482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3242.145810] alg: No test for adler32 (adler32-zlib) [ 3243.248933] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3243.567818] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 3245.223726] Key type lgssc registered [ 3246.418985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3255.006239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3255.034228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3256.524346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3261.508471] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3267.047498] LustreError: 111032:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3267.063722] LustreError: 111032:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3269.618423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3270.043163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3273.908312] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3280.975803] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3287.084314] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3293.512546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3293.844181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3301.522523] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3302.066792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 3310.903581] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3316.045190] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3319.488374] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3325.928260] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3325.945830] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3325.963478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3327.971578] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3327.978301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3327.991897] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3330.578817] Lustre: server umount lustre-OST0000 complete [ 3336.672531] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3336.695011] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3336.709742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3336.719439] Lustre: Skipped 1 previous similar message [ 3340.256913] Lustre: server umount lustre-MDT0000 complete [ 3343.338119] LustreError: 111033:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3343.361445] LustreError: 111033:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3343.516541] LustreError: 113564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442656 with bad export cookie 2145039033407111454 [ 3343.520579] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3343.533760] LustreError: 113564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3343.870467] Lustre: server umount lustre-MDT0001 complete [ 3354.768303] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 3357.645745] Key type lgssc unregistered [ 3357.899599] LNet: 114668:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3357.910327] LNetError: 114668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3357.921196] LNet: Removed LNI 192.168.203.134@tcp [ 3358.792436] Key type .llcrypt unregistered [ 3358.797348] Key type ._llcrypt unregistered [ 3378.589894] Key type ._llcrypt registered [ 3378.592166] Key type .llcrypt registered [ 3378.669086] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3379.458832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3379.549114] alg: No test for adler32 (adler32-zlib) [ 3380.804291] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3381.146738] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 3382.841679] Key type lgssc registered [ 3384.465344] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3393.001955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3393.012297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3394.510102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3399.630693] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3404.781895] LustreError: 115885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3404.808222] LustreError: 115885:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3408.435309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3408.816917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3413.363617] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3421.267522] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3426.513665] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3433.674484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3434.078978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3439.778349] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3440.247725] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3446.067445] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3464.160348] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3464.172144] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3464.187266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3464.211956] Lustre: Skipped 1 previous similar message [ 3468.815488] Lustre: server umount lustre-OST0000 complete [ 3474.403847] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3474.413328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3474.424517] Lustre: Skipped 1 previous similar message [ 3474.439777] Lustre: Skipped 2 previous similar messages [ 3478.648315] Lustre: server umount lustre-MDT0000 complete [ 3479.528959] LustreError: 118014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3479.557067] LustreError: 118014:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3482.386923] LustreError: 117730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442795 with bad export cookie 8655693470163960450 [ 3482.390217] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3482.398444] LustreError: 117730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3482.766938] Lustre: server umount lustre-MDT0001 complete [ 3493.874657] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 3496.734407] Key type lgssc unregistered [ 3497.049734] LNet: 119439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3497.063324] LNetError: 119439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3497.091939] LNet: Removed LNI 192.168.203.134@tcp [ 3498.160392] Key type .llcrypt unregistered [ 3498.164453] Key type ._llcrypt unregistered [ 3522.241596] Key type ._llcrypt registered [ 3522.244859] Key type .llcrypt registered [ 3522.340462] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3523.443386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3523.540156] alg: No test for adler32 (adler32-zlib) [ 3524.648919] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3524.953683] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 3526.650821] Key type lgssc registered [ 3528.030675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3536.275598] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3536.291583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3537.700212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3542.346938] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3548.134054] LustreError: 120858:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3548.158386] LustreError: 120858:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3550.833274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3551.168619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3554.972600] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3564.530600] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3571.261469] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3578.553376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3579.025391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3583.103221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3584.870237] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3591.513713] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3602.793648] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3607.352350] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3607.357927] Lustre: Skipped 2 previous similar messages [ 3610.208458] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3610.270715] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3612.835794] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3612.842211] Lustre: Skipped 2 previous similar messages [ 3615.618168] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3615.636870] Lustre: Skipped 1 previous similar message [ 3620.761854] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.34@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3620.768936] Lustre: Skipped 2 previous similar messages [ 3621.322512] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.34@tcp 192.168.0.[2,10]@tcp [ 3627.211725] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.34/24@tcp in log lustre-MDT0000 [ 3627.221300] Lustre: Skipped 2 previous similar messages [ 3627.591553] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.34/24@tcp [ 3627.600740] Lustre: Skipped 1 previous similar message [ 3634.656212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3634.676083] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3634.703736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3636.418523] Lustre: server umount lustre-OST0000 complete [ 3641.828571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3641.834356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3641.842724] Lustre: Skipped 1 previous similar message [ 3646.455215] Lustre: server umount lustre-MDT0000 complete [ 3646.949385] LustreError: 123348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3646.969216] LustreError: 123348:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3649.865382] LustreError: 120837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442962 with bad export cookie 9016042081616778651 [ 3649.867373] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3649.887843] LustreError: 120837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3660.601298] Lustre: server umount lustre-MDT0001 complete [ 3674.679965] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 3677.412447] Key type lgssc unregistered [ 3677.729106] LNet: 125464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3677.741619] LNetError: 125464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3677.762820] LNet: Removed LNI 192.168.203.134@tcp [ 3678.879224] Key type .llcrypt unregistered [ 3678.881977] Key type ._llcrypt unregistered [ 3709.249640] Key type ._llcrypt registered [ 3709.253288] Key type .llcrypt registered [ 3709.369444] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3710.688893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3710.883782] alg: No test for adler32 (adler32-zlib) [ 3711.991804] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3712.352753] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 3714.031171] Key type lgssc registered [ 3715.294775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3723.076617] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3723.123874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3724.513812] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3724.530394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3724.536364] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.34/24@tcp [ 3729.274843] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3735.010348] LustreError: 127021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3735.051134] LustreError: 127021:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3738.507411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3738.726903] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3738.749707] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.34/24@tcp [ 3738.824060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3744.798698] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3752.738070] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3758.045268] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3764.805176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3765.166538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3770.328901] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 3771.793394] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3780.039910] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3793.899531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3793.899837] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3793.922065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3793.938272] Lustre: Skipped 1 previous similar message [ 3799.020459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3799.029872] Lustre: Skipped 1 previous similar message [ 3799.913935] Lustre: server umount lustre-OST0000 complete [ 3804.138179] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3804.146594] Lustre: Skipped 1 previous similar message [ 3804.149794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3804.158562] Lustre: Skipped 1 previous similar message [ 3809.257262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3809.263180] Lustre: Skipped 1 previous similar message [ 3809.900503] Lustre: server umount lustre-MDT0000 complete [ 3813.231445] LustreError: 127001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443125 with bad export cookie 13263865891119013129 [ 3813.245880] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3813.247800] LustreError: 127001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3813.859158] Lustre: server umount lustre-MDT0001 complete [ 3824.644149] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 3827.734519] Key type lgssc unregistered [ 3828.108602] LNet: 130608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3828.122668] LNetError: 130608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3828.144357] LNet: Removed LNI 192.168.203.134@tcp [ 3829.137811] Key type .llcrypt unregistered [ 3829.139610] Key type ._llcrypt unregistered [ 3855.552238] Key type ._llcrypt registered [ 3855.559352] Key type .llcrypt registered [ 3855.645530] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 3873.681439] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3874.324574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3874.332694] alg: No test for adler32 (adler32-zlib) [ 3875.782572] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3876.204946] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 3877.983132] Key type lgssc registered [ 3878.954309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3914.141953] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3914.161408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3915.501426] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3915.531744] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3915.582255] Lustre: lustre-MDT0000: new disk, initializing [ 3915.687720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3915.708506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3919.900571] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3929.321285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3929.467487] Lustre: 134006:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3929.555845] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3929.558822] Lustre: Skipped 1 previous similar message [ 3929.666935] Lustre: lustre-MDT0001: new disk, initializing [ 3929.762960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3929.812201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3929.825422] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3934.527887] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3943.759292] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3949.541592] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3955.980861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3956.287295] Lustre: lustre-OST0000: new disk, initializing [ 3956.293336] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3956.300902] Lustre: 135974:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3956.409628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3962.023797] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3962.445136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3962.455124] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3962.556864] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3970.636597] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3975.041099] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3975.251234] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3979.271219] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3979.464740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3983.329265] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3983.329884] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3983.338527] LustreError: Skipped 1 previous similar message [ 3983.344410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3983.368149] Lustre: Skipped 1 previous similar message [ 3983.405200] Lustre: Skipped 1 previous similar message [ 3987.369073] Lustre: server umount lustre-OST0000 complete [ 3991.525240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3991.538138] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3991.552637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3993.568721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3993.570603] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3998.689253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3998.699962] Lustre: Skipped 2 previous similar messages [ 4003.812643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4003.826442] Lustre: Skipped 1 previous similar message [ 4005.343089] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4005.626320] Lustre: server umount lustre-MDT0000 complete [ 4010.180444] LustreError: 133999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443322 with bad export cookie 14391167712534027521 [ 4010.184735] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4010.192443] LustreError: 133999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4010.442550] Lustre: server umount lustre-MDT0001 complete [ 4018.795406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4019.271682] LustreError: 138193:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4019.290843] LustreError: 138193:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4019.349663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4023.911604] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4024.804583] LustreError: 138194:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4029.929735] LustreError: 138193:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4032.667262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4032.954975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4036.571890] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4043.229435] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4049.588952] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4056.257845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4056.456818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4062.815579] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4069.359887] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4073.622820] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4073.831382] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4078.312656] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4078.489932] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4087.141914] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4087.375924] Lustre: lustre-OST0001: new disk, initializing [ 4087.381556] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4087.387843] Lustre: 141562:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4087.515900] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4093.490955] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4094.532216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4094.547603] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4094.652582] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4101.558347] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4101.794438] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4106.144656] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4106.377077] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4111.785368] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9861045a3000.ost_server_uuid 50 [ 4113.027253] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9861045a3000.ost_server_uuid in FULL state after 0 sec [ 4120.552534] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4120.578689] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4120.579602] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4120.596916] Lustre: Skipped 1 previous similar message [ 4126.344305] Lustre: server umount lustre-OST0001 complete [ 4130.788209] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4130.797528] LustreError: 140015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4130.808591] Lustre: Skipped 2 previous similar messages [ 4130.823019] LustreError: 140015:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4135.908320] LustreError: 141863:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4135.924036] LustreError: 141863:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4136.332578] Lustre: server umount lustre-OST0000 complete [ 4140.513294] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4140.521433] LustreError: Skipped 1 previous similar message [ 4140.524619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4140.542898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4140.544815] Lustre: Skipped 6 previous similar messages [ 4145.697997] Lustre: server umount lustre-MDT0000 complete [ 4146.149812] LustreError: 138194:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4146.159681] LustreError: 138194:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4148.694635] LustreError: 138173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443461 with bad export cookie 14391167712534028403 [ 4148.697233] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4148.707195] LustreError: 138173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4149.011467] Lustre: server umount lustre-MDT0001 complete [ 4166.696779] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 4169.129517] Key type lgssc unregistered [ 4169.399937] LNet: 144567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4169.413611] LNetError: 144567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4169.445915] LNet: Removed LNI 192.168.203.134@tcp [ 4170.336580] Key type .llcrypt unregistered [ 4170.340235] Key type ._llcrypt unregistered [ 4204.465350] Key type ._llcrypt registered [ 4204.467461] Key type .llcrypt registered [ 4204.586405] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 4218.953930] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 4220.267227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4220.328853] alg: No test for adler32 (adler32-zlib) [ 4221.679862] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4222.255903] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 4223.999330] Key type lgssc registered [ 4226.003873] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4262.796989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4262.828142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4264.063572] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4264.134793] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4264.272751] Lustre: lustre-MDT0000: new disk, initializing [ 4264.400212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4264.428037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4268.696934] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4282.183031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4282.295800] Lustre: 148713:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4282.369323] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4282.378460] Lustre: Skipped 1 previous similar message [ 4282.460983] Lustre: lustre-MDT0001: new disk, initializing [ 4282.560834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4282.604853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4282.622165] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4286.900678] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4296.000479] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4300.545579] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4306.562810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4306.833314] Lustre: lustre-OST0000: new disk, initializing [ 4306.838275] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4306.847310] Lustre: 150681:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4306.995953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4313.632734] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4316.772286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4316.789691] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4316.946110] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4323.663593] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4327.898491] Lustre: Failing over lustre-OST0000 [ 4328.125821] Lustre: server umount lustre-OST0000 complete [ 4345.823319] Lustre: 146936:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443642/real 1788443642] req@00000000a81be9dd x1875318967140096/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788443658 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4345.881991] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4347.436267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4347.807723] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4347.813623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4347.860071] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4348.383114] Lustre: 146937:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443645/real 1788443645] req@000000002b905071 x1875318967141248/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788443661 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4350.825233] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4353.505692] Lustre: 146936:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443650/real 1788443650] req@0000000078d692e5 x1875318967142144/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788443666 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4353.561933] Lustre: 146936:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4355.206447] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4357.447971] Lustre: lustre-OST0000: Recovery over after 0:07, of 3 clients 3 recovered and 0 were evicted. [ 4357.447977] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4358.495846] Lustre: 146937:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443655/real 1788443655] req@000000008d411511 x1875318967143296/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788443671 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4358.534671] Lustre: 146937:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4359.029848] Lustre: Failing over lustre-MDT0000 [ 4359.288579] Lustre: server umount lustre-MDT0000 complete [ 4359.649370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4359.657763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4366.312391] LustreError: 148723:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4366.325042] LustreError: 148723:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 4367.653945] LustreError: 148720:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4371.430710] LustreError: 148724:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4371.460472] LustreError: 148724:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4376.544159] Lustre: 146938:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443673/real 1788443673] req@000000009b94e2a3 x1875318967153408/t0(0) o400->MGC192.168.203.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788443689 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4376.545016] LustreError: 148723:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4376.566684] Lustre: 146938:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4376.582810] LustreError: 148723:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4376.593162] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4379.086099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4381.667160] LustreError: 148720:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4381.684498] LustreError: 148720:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4386.804201] LustreError: 146935:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000334bfb41 x1875318967158784/t0(0) o250->MGC192.168.203.134@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4387.388771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4387.464900] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4389.130371] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4392.463687] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4392.478685] Lustre: Skipped 1 previous similar message [ 4392.510726] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4392.607328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4393.588700] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4403.169846] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4403.181520] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4403.206053] Lustre: Skipped 2 previous similar messages [ 4403.220764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4407.779328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4407.792261] Lustre: Skipped 2 previous similar messages [ 4409.175551] Lustre: server umount lustre-OST0000 complete [ 4418.018819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4418.020364] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4418.036949] LustreError: Skipped 1 previous similar message [ 4418.043424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4418.093020] Lustre: Skipped 2 previous similar messages [ 4423.150229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4423.159562] Lustre: Skipped 2 previous similar messages [ 4428.257720] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4428.260441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4428.271761] Lustre: Skipped 1 previous similar message [ 4428.449212] Lustre: server umount lustre-MDT0000 complete [ 4432.354364] LustreError: 148723:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4432.367215] LustreError: 148723:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4432.499092] LustreError: 148705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443745 with bad export cookie 16502059510060946270 [ 4432.502486] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4432.532122] LustreError: 148705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4432.887297] Lustre: server umount lustre-MDT0001 complete [ 4443.678977] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 4447.809575] Key type lgssc unregistered [ 4448.356608] LNet: 154335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4448.373597] LNetError: 154335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4448.402171] LNet: Removed LNI 192.168.203.134@tcp [ 4449.337685] Key type .llcrypt unregistered [ 4449.342798] Key type ._llcrypt unregistered [ 4490.732518] Key type ._llcrypt registered [ 4490.736224] Key type .llcrypt registered [ 4490.886925] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 4491.786718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4491.818846] alg: No test for adler32 (adler32-zlib) [ 4492.929674] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4493.161492] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 4494.824568] Key type lgssc registered [ 4495.833561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4503.846871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4503.864440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4505.370660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4509.662182] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4515.824771] LustreError: 155853:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4515.854079] LustreError: 155853:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4519.191800] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4520.929947] LustreError: 155854:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4523.367086] LustreError: 155882:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4523.373940] LustreError: 155882:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4523.378482] LustreError: 155882:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4523.590817] Lustre: server umount lustre-MDT0000 complete [ 4539.855880] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 4551.613785] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 4587.749578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4587.969454] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4588.019339] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4588.107688] Lustre: lustre-MDT0000: new disk, initializing [ 4588.180365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4588.194491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4593.188448] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4604.149273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4604.259368] Lustre: 159187:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4604.334861] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4604.339191] Lustre: Skipped 1 previous similar message [ 4604.448107] Lustre: lustre-MDT0001: new disk, initializing [ 4604.542097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4604.572796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4604.587831] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4610.027837] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4619.369347] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4625.159684] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4632.828671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4633.056372] Lustre: lustre-OST0000: new disk, initializing [ 4633.066052] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4633.072873] Lustre: 161155:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4633.148539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4638.651762] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4640.889602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4640.910765] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4641.000941] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4650.191954] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4656.463011] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4658.524148] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 09:59:28 (1788443968) [ 4921.824047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4921.835116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4921.862665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4922.847510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4922.848319] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4922.875388] Lustre: Skipped 1 previous similar message [ 4922.908087] Lustre: Skipped 1 previous similar message [ 4927.968607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4927.984509] Lustre: Skipped 3 previous similar messages [ 4933.099862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4933.104331] Lustre: Skipped 2 previous similar messages [ 4935.135130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4935.557926] Lustre: server umount lustre-MDT0000 complete [ 4938.210897] LustreError: 159196:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4938.233818] LustreError: 159196:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4939.636719] LustreError: 159181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444252 with bad export cookie 13582920872142989652 [ 4939.637582] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4939.648448] LustreError: 159181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4943.328076] LustreError: 162962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4943.328357] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4943.354570] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4945.919469] Lustre: server umount lustre-MDT0001 complete [ 4955.705307] Lustre: server umount lustre-OST0000 complete [ 4965.253924] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 4975.544735] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 5013.562603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5013.930593] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5013.977520] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5014.095511] Lustre: lustre-MDT0000: new disk, initializing [ 5014.254373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5014.278335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5018.799978] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5029.179306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5029.456569] Lustre: 165837:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5029.604665] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5029.611551] Lustre: Skipped 1 previous similar message [ 5029.672586] Lustre: lustre-MDT0001: new disk, initializing [ 5029.756880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5029.785212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5029.795848] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5033.593115] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5044.011587] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5051.092535] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5058.921687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5059.433459] Lustre: lustre-OST0000: new disk, initializing [ 5059.443421] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5059.461653] Lustre: 167803:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5059.591732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5064.821048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5064.831564] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5064.994370] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5067.745418] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5078.509477] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5084.579493] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5084.979780] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5089.599885] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5089.798470] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5095.904626] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5095.907182] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5095.925310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5095.925319] Lustre: Skipped 1 previous similar message [ 5095.958425] LustreError: Skipped 1 previous similar message [ 5097.761447] Lustre: server umount lustre-OST0000 complete [ 5106.151397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5106.169241] Lustre: Skipped 1 previous similar message [ 5107.983500] Lustre: server umount lustre-MDT0000 complete [ 5111.268095] LustreError: 165847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5111.293825] LustreError: 165847:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5111.902402] LustreError: 165828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444424 with bad export cookie 13582920872143148482 [ 5111.908075] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5111.911409] LustreError: 165828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5112.327459] Lustre: server umount lustre-MDT0001 complete [ 5121.272667] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:07:11 (1788444431) [ 5138.426402] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 5149.938743] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 5187.947701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5188.116439] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5188.156263] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5188.234676] Lustre: lustre-MDT0000: new disk, initializing [ 5188.343281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5188.370145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5193.356286] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5203.306462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5203.457436] Lustre: 172560:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5203.554530] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5203.561221] Lustre: Skipped 3 previous similar messages [ 5203.810234] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5209.066173] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5218.970191] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5225.184916] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5231.962718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5232.334098] Lustre: lustre-OST0000: new disk, initializing [ 5232.337543] Lustre: Skipped 1 previous similar message [ 5232.340795] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5232.348872] Lustre: Skipped 2 previous similar messages [ 5232.355456] Lustre: 174539:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5239.147448] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5241.924143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5241.931536] Lustre: Skipped 1 previous similar message [ 5241.938593] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5242.072699] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5248.243741] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5257.701235] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5257.712242] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5257.720139] Lustre: Skipped 1 previous similar message [ 5257.724749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5257.728834] Lustre: Skipped 1 previous similar message [ 5262.507519] Lustre: server umount lustre-OST0000 complete [ 5281.253581] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5281.506833] Lustre: server umount lustre-MDT0000 complete [ 5283.297839] LustreError: 172568:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5283.331168] LustreError: 172568:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5285.236365] LustreError: 172553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444597 with bad export cookie 13582920872143149364 [ 5285.240669] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5285.247067] LustreError: 172553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5285.579539] Lustre: server umount lustre-MDT0001 complete [ 5294.368609] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:10:04 (1788444604) [ 5310.959830] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 5322.162676] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 5358.712290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5358.982794] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5359.015381] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5359.093117] Lustre: lustre-MDT0000: new disk, initializing [ 5359.206962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5359.213427] Lustre: Skipped 2 previous similar messages [ 5359.231479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5364.156853] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5374.976096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5375.185908] Lustre: 178867:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5375.518443] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5379.850629] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5388.664640] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5394.598894] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5401.244906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5401.494429] Lustre: 180843:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5408.026616] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5410.362223] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5410.476919] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5418.002629] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5426.146255] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5426.148104] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5426.160575] LustreError: Skipped 2 previous similar messages [ 5426.163588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5426.163596] Lustre: Skipped 6 previous similar messages [ 5426.212171] Lustre: Skipped 4 previous similar messages [ 5432.147989] Lustre: server umount lustre-OST0000 complete [ 5443.570257] Lustre: server umount lustre-MDT0000 complete [ 5446.631964] LustreError: 178879:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5446.678325] LustreError: 178879:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5447.648956] LustreError: 178860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444760 with bad export cookie 13582920872143150449 [ 5447.655390] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5447.675980] LustreError: 178860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5447.931908] Lustre: server umount lustre-MDT0001 complete [ 5459.517682] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 5462.480950] Key type lgssc unregistered [ 5462.807783] LNet: 182697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5462.812335] LNetError: 182697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5463.854710] LNet: Removed LNI 192.168.203.134@tcp [ 5464.958070] Key type .llcrypt unregistered [ 5464.962632] Key type ._llcrypt unregistered [ 5490.603912] Key type ._llcrypt registered [ 5490.606676] Key type .llcrypt registered [ 5490.773164] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 5492.076252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5492.143398] alg: No test for adler32 (adler32-zlib) [ 5493.370839] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5493.640035] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 5495.351242] Key type lgssc registered [ 5496.806467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5505.724943] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5505.744278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5507.210172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5511.864462] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5516.789942] LustreError: 184116:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5516.812494] LustreError: 184116:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5520.685388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5521.126448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5525.705270] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5533.239373] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5539.254415] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5545.728520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5545.892830] LustreError: 185938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5546.148838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5551.318937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5553.190367] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5559.819603] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5582.305875] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5582.308868] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5582.325293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5582.341030] Lustre: Skipped 1 previous similar message [ 5585.718735] Lustre: server umount lustre-OST0000 complete [ 5592.554909] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5592.573179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5592.580985] Lustre: Skipped 2 previous similar messages [ 5596.087980] Lustre: server umount lustre-MDT0000 complete [ 5597.665214] LustreError: 184116:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5597.682310] LustreError: 184116:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5600.407338] LustreError: 184097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444913 with bad export cookie 10813521251299344962 [ 5600.418363] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5600.994785] Lustre: server umount lustre-MDT0001 complete [ 5614.708659] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 5617.931768] Key type lgssc unregistered [ 5618.173647] LNet: 187672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5618.180983] LNetError: 187672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5618.200938] LNet: Removed LNI 192.168.203.134@tcp [ 5619.462056] Key type .llcrypt unregistered [ 5619.465209] Key type ._llcrypt unregistered [ 5647.737120] Key type ._llcrypt registered [ 5647.740169] Key type .llcrypt registered [ 5647.838327] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 5649.358289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5649.478122] alg: No test for adler32 (adler32-zlib) [ 5650.582877] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5650.813981] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 5652.495171] Key type lgssc registered [ 5654.115618] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5663.405803] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5663.415670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5665.089693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5670.079598] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5674.473236] LustreError: 189085:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5674.500635] LustreError: 189085:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5679.390756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5679.587230] LustreError: 189086:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5679.939808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5684.420787] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5692.174891] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5698.015709] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5704.285779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5704.673982] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 5704.884678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5710.018887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5710.893447] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5718.149123] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5726.003911] Lustre: lustre-OST0000: Not available for connect from 192.168.203.34@tcp (stopping) [ 5728.751469] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5728.757763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5728.786552] Lustre: Skipped 1 previous similar message [ 5728.804563] Lustre: Skipped 1 previous similar message [ 5731.080470] Lustre: lustre-OST0000: Not available for connect from 192.168.203.34@tcp (stopping) [ 5736.205309] Lustre: lustre-OST0000: Not available for connect from 192.168.203.34@tcp (stopping) [ 5736.218213] Lustre: Skipped 2 previous similar messages [ 5739.999203] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5740.277775] Lustre: server umount lustre-OST0000 complete [ 5748.747432] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9861056d5000.ost_server_uuid 50 [ 5750.507923] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5767.137270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5767.147715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5767.175568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5767.188432] Lustre: Skipped 2 previous similar messages [ 5769.273058] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5771.333211] Lustre: server umount lustre-MDT0000 complete [ 5774.307317] LustreError: 190099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5774.333751] LustreError: 190099:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5775.206151] LustreError: 189065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445087 with bad export cookie 17009956359580623476 [ 5775.213725] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5775.227278] LustreError: 189065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5775.610490] Lustre: server umount lustre-MDT0001 complete [ 5784.181809] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:18:14 (1788445094) [ 5790.190431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5790.773714] LustreError: 193003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5790.861330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5795.978509] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5796.332905] LustreError: 193004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5805.543874] LustreError: 193003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5805.573594] LustreError: 193003:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5805.872445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5806.137693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5811.613434] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5819.071765] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5824.049873] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5830.764122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5831.107062] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5836.292665] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5838.187095] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5846.688360] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5854.358474] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5854.649682] Lustre: lustre-OST0001: new disk, initializing [ 5854.656306] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5854.676677] Lustre: 195847:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5854.848876] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5861.118905] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5864.536869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5864.562361] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5864.695858] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5871.938959] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5876.199470] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5876.216811] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5876.244557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5876.257644] Lustre: Skipped 2 previous similar messages [ 5877.217080] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5881.550534] Lustre: server umount lustre-OST0000 complete [ 5883.162819] LustreError: 194828:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5889.263495] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5889.582076] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5894.896390] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5895.146777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5911.010326] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5911.030364] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5911.038111] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5911.047759] Lustre: Skipped 2 previous similar messages [ 5911.060250] Lustre: Skipped 5 previous similar messages [ 5915.939124] Lustre: server umount lustre-OST0001 complete [ 5921.250237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5921.268178] Lustre: Skipped 1 previous similar message [ 5926.482187] Lustre: server umount lustre-MDT0000 complete [ 5929.999879] LustreError: 192985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445242 with bad export cookie 17009956359580624547 [ 5930.007847] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5930.016014] LustreError: 192985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5930.444582] Lustre: server umount lustre-MDT0001 complete [ 5953.003430] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:21:03 (1788445263) [ 5959.109551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5959.530340] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5959.572141] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5959.905533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5965.606287] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5974.072581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5974.293484] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5974.407535] Lustre: 199284:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5974.540406] LustreError: 199296:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5974.564573] LustreError: 199296:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [ 5980.821780] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5988.261470] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5994.611208] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6000.573886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6000.802932] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6006.950779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6007.679691] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6015.287198] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6022.038769] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6022.145165] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6029.879501] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6030.411374] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6038.835348] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6042.594270] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6042.603083] LustreError: Skipped 1 previous similar message [ 6042.613618] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6042.646879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6042.653990] Lustre: Skipped 4 previous similar messages [ 6047.647820] Lustre: server umount lustre-OST0000 complete [ 6049.560825] LustreError: 201136:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6066.155777] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6066.163692] LustreError: Skipped 1 previous similar message [ 6071.198103] Lustre: server umount lustre-OST0001 complete [ 6076.385563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6076.409324] Lustre: Skipped 4 previous similar messages [ 6081.004507] Lustre: server umount lustre-MDT0000 complete [ 6084.623546] LustreError: 199277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445397 with bad export cookie 17009956359580625954 [ 6084.646823] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6084.928483] Lustre: server umount lustre-MDT0001 complete [ 6108.983084] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:23:38 (1788445418) [ 6127.435426] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 6138.376047] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 6175.379742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6175.561827] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6175.586888] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6175.675207] Lustre: lustre-MDT0000: new disk, initializing [ 6175.725890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6175.731805] Lustre: Skipped 3 previous similar messages [ 6175.759852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6182.247335] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6193.125700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6193.324662] Lustre: 207385:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6193.425268] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6193.435482] Lustre: Skipped 1 previous similar message [ 6193.581500] Lustre: lustre-MDT0001: new disk, initializing [ 6193.759134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6193.770742] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6198.454975] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6208.078377] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6213.317721] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6220.016825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6220.369603] Lustre: lustre-OST0000: new disk, initializing [ 6220.379538] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6220.403406] Lustre: 209353:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6227.109255] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6230.595341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6230.605280] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6230.705320] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6237.422938] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6242.973640] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6243.221390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6247.724764] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6247.988829] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6251.494938] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6251.498692] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6251.510883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6251.519186] Lustre: Skipped 9 previous similar messages [ 6256.483313] Lustre: server umount lustre-OST0000 complete [ 6267.043770] Lustre: server umount lustre-MDT0000 complete [ 6271.488039] LustreError: 207375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445584 with bad export cookie 17009956359580627739 [ 6271.506470] LustreError: 207375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6271.507395] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6271.888876] Lustre: server umount lustre-MDT0001 complete [ 6286.315608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6286.933078] LustreError: 211569:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6286.962522] LustreError: 211569:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 6292.543750] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6302.308060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6307.603398] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6315.480449] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6321.572270] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6328.206043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6328.679247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6328.688327] Lustre: Skipped 4 previous similar messages [ 6335.207429] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6341.796304] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6346.470481] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6346.659593] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6350.443641] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6350.641246] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6353.381047] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6353.387797] LustreError: Skipped 1 previous similar message [ 6354.878915] Lustre: server umount lustre-OST0000 complete [ 6361.824430] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6366.522413] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6396.437533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6404.425894] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6413.790933] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6415.719958] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6419.244667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6419.268769] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6419.270078] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6419.292097] Lustre: Skipped 1 previous similar message [ 6424.550647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6424.578431] Lustre: Skipped 5 previous similar messages [ 6424.588810] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6426.678232] Lustre: server umount lustre-OST0000 complete [ 6436.729559] Lustre: server umount lustre-MDT0000 complete [ 6440.656387] LustreError: 211551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445753 with bad export cookie 17009956359580628621 [ 6440.665536] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6440.669169] LustreError: 211551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6441.254898] Lustre: server umount lustre-MDT0001 complete [ 6450.374359] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:29:20 (1788445760) [ 6456.546788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6461.932945] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6469.896832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6474.215989] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6481.779973] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6487.920533] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6494.482679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6500.887960] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6509.622912] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6514.733914] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6514.915341] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6519.758529] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6519.983767] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6526.677201] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6526.936883] Lustre: lustre-OST0001: new disk, initializing [ 6526.940660] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6526.961932] Lustre: 220905:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6533.836532] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6534.747292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6534.764853] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6534.869367] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6543.183317] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6546.918902] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6547.105568] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6551.421563] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6551.693781] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6555.617150] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6555.639290] LustreError: Skipped 2 previous similar messages [ 6555.656887] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6555.674438] Lustre: Skipped 12 previous similar messages [ 6559.740939] Lustre: server umount lustre-OST0001 complete [ 6560.740125] LustreError: 219957:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6560.763175] LustreError: 219957:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 6566.022410] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6566.285877] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6571.117317] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6571.347637] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6603.190764] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6603.650167] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6603.669265] Lustre: Skipped 5 previous similar messages [ 6608.869459] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6608.901434] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6608.902163] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6608.925426] Lustre: Skipped 1 previous similar message [ 6612.374669] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6620.942662] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6623.018782] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6631.470962] Lustre: server umount lustre-OST0001 complete [ 6663.136557] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6663.369396] Lustre: server umount lustre-OST0000 complete [ 6673.556314] Lustre: server umount lustre-MDT0000 complete [ 6677.151742] LustreError: 217517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445989 with bad export cookie 17009956359580629587 [ 6677.162075] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6677.176795] LustreError: 217517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6683.609565] Lustre: server umount lustre-MDT0001 complete [ 6709.909785] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:33:40 (1788446020) [ 6716.597569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6716.992119] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6717.048102] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6721.481580] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6729.341690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6729.542090] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6729.580012] Lustre: 226153:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6734.162455] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6741.569337] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6747.950915] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6755.319664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6755.627313] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6762.249198] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6772.118132] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6782.344638] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6782.425790] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6789.969843] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6797.875756] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6799.401740] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6804.811864] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6805.084992] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6809.708703] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6809.964679] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6815.807401] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98610538d000.ost_server_uuid 50 [ 6817.337811] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff98610538d000.ost_server_uuid in IDLE state after 0 sec [ 6818.543256] Lustre: Permanently deactivating lustre-OST0001 [ 6818.577320] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6822.334631] Lustre: Permanently reactivating lustre-OST0001 [ 6829.327396] Lustre: server umount lustre-OST0001 complete [ 6831.074477] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6831.078964] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6831.088374] LustreError: Skipped 7 previous similar messages [ 6831.106538] Lustre: Skipped 11 previous similar messages [ 6839.378486] Lustre: server umount lustre-OST0000 complete [ 6849.470950] Lustre: server umount lustre-MDT0000 complete [ 6854.009921] LustreError: 226144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446166 with bad export cookie 17009956359580630875 [ 6854.022498] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6854.031889] LustreError: 226144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6854.339349] Lustre: server umount lustre-MDT0001 complete [ 6880.577823] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:36:30 (1788446190) [ 6888.882163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6889.180907] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6889.211255] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6889.217760] Lustre: Skipped 5 previous similar messages [ 6894.797445] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6903.513660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6903.594122] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6903.625274] Lustre: 232846:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6908.850466] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6916.654586] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6922.870506] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6930.055928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6930.336737] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6930.386259] Lustre: Permanently deactivating lustre-OST0000 [ 6930.424553] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6937.881671] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6940.747819] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6945.926353] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6953.361937] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6953.469502] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6958.812419] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6960.304611] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6968.124507] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6969.907722] Lustre: Permanently reactivating lustre-OST0000 [ 6969.931863] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6969.943984] Lustre: Skipped 2 previous similar messages [ 6979.638600] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6979.653736] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 39 seconds. I think it's dead, and I am evicting it. exp 00000000f9540266, cur 1788446292 deadline 1788446283 last 1788446253 [ 6979.737117] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6991.832626] Lustre: server umount lustre-OST0001 complete [ 7010.271160] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7010.583750] Lustre: server umount lustre-OST0000 complete [ 7020.501750] Lustre: server umount lustre-MDT0000 complete [ 7023.703700] LustreError: 232840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446336 with bad export cookie 17009956359580632548 [ 7023.711088] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7023.713480] LustreError: 232840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7024.015908] Lustre: server umount lustre-MDT0001 complete [ 7034.227340] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 7037.130733] Key type lgssc unregistered [ 7037.416720] LNet: 237726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7037.425880] LNetError: 237726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7037.447043] LNet: Removed LNI 192.168.203.134@tcp [ 7038.192908] Key type .llcrypt unregistered [ 7038.196311] Key type ._llcrypt unregistered [ 7058.487206] Key type ._llcrypt registered [ 7058.493251] Key type .llcrypt registered [ 7058.603805] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 7059.379763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7059.432654] alg: No test for adler32 (adler32-zlib) [ 7060.427288] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7060.625028] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 7062.255856] Key type lgssc registered [ 7063.395362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7076.320625] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7076.346743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7078.067613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7083.802580] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7087.605280] LustreError: 239183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7087.638960] LustreError: 239183:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7091.734323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7091.871088] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7091.897757] Lustre: 239170:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7091.907550] Lustre: Permanently deactivating lustre-MDT0001 [ 7091.919275] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7092.170808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7094.474507] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7094.483137] LustreError: 239213:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7094.500369] LustreError: 239213:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7094.517035] LustreError: 239213:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 1, failed: rc = -108 [ 7095.895222] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7102.756180] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7107.608903] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7113.496103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7113.867743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7116.921976] LustreError: 241026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7120.005758] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7122.407655] LustreError: 241024:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7123.440611] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7127.299510] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7128.555522] LustreError: 241203:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7128.588238] LustreError: 241203:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7133.685971] LustreError: 241026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7133.709207] LustreError: 241026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7134.087909] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7134.394422] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7139.829806] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7139.843038] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7140.740258] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7147.642636] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7150.328282] Lustre: Permanently reactivating lustre-MDT0001 [ 7150.350509] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7150.361629] Lustre: Skipped 2 previous similar messages [ 7153.466032] Lustre: lustre-MDT0001: Client 96c815ca-3732-44e7-9cec-418f0432beaf (at 192.168.203.34@tcp) reconnecting [ 7159.493548] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7161.036763] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7166.368789] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7166.599692] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7171.003442] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7171.286913] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7173.462879] Lustre: Permanently deactivating lustre-MDT0001 [ 7173.474795] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7173.482806] Lustre: Skipped 2 previous similar messages [ 7179.346497] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7187.179057] Lustre: server umount lustre-MDT0000 complete [ 7189.478809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7189.486692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7189.501266] LustreError: 243677:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7189.991171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7190.010346] Lustre: Skipped 2 previous similar messages [ 7190.788189] LustreError: 241045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446503 with bad export cookie 13750420361359316818 [ 7190.796762] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7190.808402] LustreError: 241045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7191.162387] Lustre: server umount lustre-MDT0001 complete [ 7194.590988] Lustre: server umount lustre-OST0000 complete [ 7204.902214] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:41:55 (1788446515) [ 7214.716641] Lustre: server umount lustre-OST0001 complete [ 7220.831460] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 7231.529836] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 7266.256051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7266.507282] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7266.514236] Lustre: Skipped 2 previous similar messages [ 7266.533892] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7266.687626] Lustre: lustre-MDT0000: new disk, initializing [ 7266.797859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7266.850773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7271.231716] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7280.588491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7280.697529] Lustre: 247440:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7280.742445] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7280.745494] Lustre: Skipped 1 previous similar message [ 7280.808966] Lustre: lustre-MDT0001: new disk, initializing [ 7280.908156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7280.968969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7280.989404] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7285.310576] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7294.733548] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7300.143477] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7306.687493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7306.959448] Lustre: lustre-OST0000: new disk, initializing [ 7306.963774] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7306.971349] Lustre: 249406:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7307.069362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7313.200836] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7315.522867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7315.542731] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7315.657608] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7321.724613] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7323.652436] LustreError: 247447:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7331.993216] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7332.149234] Lustre: lustre-OST0001: new disk, initializing [ 7332.155301] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7332.159399] Lustre: 250521:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7333.754126] LustreError: 247447:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7333.787292] LustreError: 247446:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7338.083300] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7342.662742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7342.667732] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7342.735360] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7343.865124] LustreError: 247446:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7345.820626] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7346.819932] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7350.066393] Lustre: lustre-OST0001: Not available for connect from 192.168.203.34@tcp (stopping) [ 7353.316343] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7353.327616] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7353.341648] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7353.343243] Lustre: Skipped 1 previous similar message [ 7353.364674] Lustre: Skipped 1 previous similar message [ 7354.527331] Lustre: server umount lustre-OST0001 complete [ 7355.147217] LustreError: 249872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7355.180914] LustreError: 249872:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7372.258801] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7372.265495] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7372.268662] LustreError: Skipped 2 previous similar messages [ 7372.272730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7372.288687] Lustre: Skipped 1 previous similar message [ 7372.303669] Lustre: Skipped 1 previous similar message [ 7377.357232] Lustre: server umount lustre-OST0000 complete [ 7383.520431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7383.529182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7383.555423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7383.567599] Lustre: Skipped 2 previous similar messages [ 7388.062455] Lustre: server umount lustre-MDT0000 complete [ 7389.160969] LustreError: 247450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7389.180288] LustreError: 247450:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 7392.673601] LustreError: 247433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446705 with bad export cookie 13750420361359319415 [ 7392.679283] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7392.685376] LustreError: 247433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7399.035725] Lustre: server umount lustre-MDT0001 complete [ 7410.751427] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 7413.775549] Key type lgssc unregistered [ 7414.125940] LNet: 252553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7414.136223] LNetError: 252553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7414.166074] LNet: Removed LNI 192.168.203.134@tcp [ 7415.257276] Key type .llcrypt unregistered [ 7415.259976] Key type ._llcrypt unregistered [ 7450.736323] Key type ._llcrypt registered [ 7450.738648] Key type .llcrypt registered [ 7450.855777] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 7451.729404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7451.860030] alg: No test for adler32 (adler32-zlib) [ 7453.002859] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7453.261773] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 7454.959942] Key type lgssc registered [ 7455.943969] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7462.588601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7462.604713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7463.840626] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7463.865099] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7464.058872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7468.195555] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7474.518565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7474.630104] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7474.663544] Lustre: 254758:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7474.843955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7478.599282] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7485.054971] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7489.904531] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7495.661651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7495.808023] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7496.069525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7501.428725] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7503.483092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7507.787894] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7564.771448] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7564.780215] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7564.804319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7565.281041] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7565.289448] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7565.303869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7565.312506] Lustre: Skipped 1 previous similar message [ 7567.174705] Lustre: server umount lustre-OST0000 complete [ 7570.919087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7583.906898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7584.269500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7585.335334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7585.360476] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7590.509330] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7590.522853] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7590.967344] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7598.155432] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7636.963884] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7636.964746] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7636.975554] LustreError: Skipped 1 previous similar message [ 7636.978074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7636.991604] Lustre: Skipped 1 previous similar message [ 7637.023463] Lustre: Skipped 1 previous similar message [ 7640.037441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7640.045100] Lustre: Skipped 1 previous similar message [ 7642.086153] Lustre: server umount lustre-OST0000 complete [ 7649.248931] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7649.273217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7649.299811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7651.837682] Lustre: server umount lustre-MDT0000 complete [ 7655.584856] LustreError: 254750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446968 with bad export cookie 13408304634855566437 [ 7655.588596] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7655.596462] LustreError: 254750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7656.030822] Lustre: server umount lustre-MDT0001 complete [ 7669.116401] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 7671.470780] Key type lgssc unregistered [ 7671.656505] LNet: 260178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7671.663642] LNetError: 260178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7671.674950] LNet: Removed LNI 192.168.203.134@tcp [ 7672.523496] Key type .llcrypt unregistered [ 7672.528345] Key type ._llcrypt unregistered [ 7694.774907] Key type ._llcrypt registered [ 7694.778456] Key type .llcrypt registered [ 7694.868381] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 7696.018326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7696.052034] alg: No test for adler32 (adler32-zlib) [ 7697.155386] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7697.383745] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 7699.063178] Key type lgssc registered [ 7700.281656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7707.600990] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7707.631094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7709.197422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7713.524531] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7718.374548] LustreError: 261591:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7718.409690] LustreError: 261591:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7721.332403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7721.666610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7726.047961] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7733.008759] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7738.272885] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7744.508265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7744.793779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7750.625548] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7754.044259] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7758.170502] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7779.818793] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7779.823679] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7779.830502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7779.862835] Lustre: Skipped 1 previous similar message [ 7784.552410] Lustre: server umount lustre-OST0000 complete [ 7788.518828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7788.537414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7788.570286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7788.588652] Lustre: Skipped 1 previous similar message [ 7790.052138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7790.053289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7794.070450] Lustre: server umount lustre-MDT0000 complete [ 7795.171555] LustreError: 262639:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7795.192356] LustreError: 262639:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7797.453081] LustreError: 261572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447110 with bad export cookie 7546678931723063112 [ 7797.458938] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7797.470237] LustreError: 261572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7797.793971] Lustre: server umount lustre-MDT0001 complete [ 7808.480484] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 7811.156868] Key type lgssc unregistered [ 7811.442793] LNet: 265898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7811.454310] LNetError: 265898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7811.468273] LNet: Removed LNI 192.168.203.134@tcp [ 7812.328440] Key type .llcrypt unregistered [ 7812.331838] Key type ._llcrypt unregistered [ 7829.762234] Key type ._llcrypt registered [ 7829.764337] Key type .llcrypt registered [ 7829.840556] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 7830.689447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7830.724550] alg: No test for adler32 (adler32-zlib) [ 7831.761750] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7831.945233] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 7833.591183] Key type lgssc registered [ 7834.376486] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7841.249993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7841.291401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7842.790440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7847.949585] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7853.026030] LustreError: 267119:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7853.052622] LustreError: 267119:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7857.255667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7857.848858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7863.084986] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7869.250854] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7874.179277] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7880.387929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7880.729207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7886.499716] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7887.925864] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7893.550203] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7908.321152] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7908.321491] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7908.327029] LustreError: Skipped 1 previous similar message [ 7908.335273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7908.359762] Lustre: Skipped 1 previous similar message [ 7908.398245] Lustre: Skipped 1 previous similar message [ 7910.905157] Lustre: server umount lustre-OST0000 complete [ 7918.560880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7918.579874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7918.580634] Lustre: Skipped 1 previous similar message [ 7918.603434] Lustre: Skipped 1 previous similar message [ 7920.702153] Lustre: server umount lustre-MDT0000 complete [ 7923.684517] LustreError: 267846:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7924.243977] LustreError: 267100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447236 with bad export cookie 264657883983136620 [ 7924.251953] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7924.254632] LustreError: 267100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7924.640804] Lustre: server umount lustre-MDT0001 complete [ 7934.312935] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 7937.275050] Key type lgssc unregistered [ 7937.610643] LNet: 270795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7937.618170] LNetError: 270795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7937.631969] LNet: Removed LNI 192.168.203.134@tcp [ 7938.622409] Key type .llcrypt unregistered [ 7938.627890] Key type ._llcrypt unregistered [ 7962.432914] Key type ._llcrypt registered [ 7962.435312] Key type .llcrypt registered [ 7962.538485] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 7963.675940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7963.716754] alg: No test for adler32 (adler32-zlib) [ 7964.778252] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7965.052449] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 7966.791176] Key type lgssc registered [ 7967.915647] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7975.901361] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7975.942027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7977.390666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7981.842632] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7987.684186] LustreError: 272209:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7987.720892] LustreError: 272209:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7990.106481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7990.374268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7994.620722] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8001.427447] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8006.646226] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8012.923479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8013.187965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8019.088575] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8022.401200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8025.838381] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8053.227403] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8053.231226] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8053.244743] Lustre: Skipped 1 previous similar message [ 8053.251864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8053.259077] Lustre: Skipped 1 previous similar message [ 8058.340643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8058.346849] Lustre: Skipped 1 previous similar message [ 8059.484391] Lustre: server umount lustre-OST0000 complete [ 8063.459216] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8063.463871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8063.469953] Lustre: Skipped 1 previous similar message [ 8063.480717] Lustre: Skipped 1 previous similar message [ 8068.578560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8068.592539] Lustre: Skipped 1 previous similar message [ 8073.697513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8073.710046] Lustre: Skipped 1 previous similar message [ 8077.279186] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8077.546989] Lustre: server umount lustre-MDT0000 complete [ 8078.816988] LustreError: 272209:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8078.831989] LustreError: 272209:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8081.472650] LustreError: 272190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447394 with bad export cookie 6534652408814116474 [ 8081.481695] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8081.493231] LustreError: 272190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8081.811493] Lustre: server umount lustre-MDT0001 complete [ 8092.886349] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 8095.847921] Key type lgssc unregistered [ 8096.214656] LNet: 276514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8096.223571] LNetError: 276514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8097.257459] LNet: Removed LNI 192.168.203.134@tcp [ 8098.485910] Key type .llcrypt unregistered [ 8098.493536] Key type ._llcrypt unregistered [ 8115.475717] Key type ._llcrypt registered [ 8115.477447] Key type .llcrypt registered [ 8115.573381] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 8116.585344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8116.605858] alg: No test for adler32 (adler32-zlib) [ 8117.715273] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8118.006325] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 8119.687150] Key type lgssc registered [ 8120.698504] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8127.674243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8127.684716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8129.337030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8134.323858] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8139.792402] LustreError: 277752:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8139.816207] LustreError: 277752:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8144.871722] LustreError: 277754:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8145.624303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8146.196590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8151.736898] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8158.873134] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8164.942271] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8171.335612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8171.610845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8174.709479] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8177.135475] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8184.286628] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8200.674677] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8200.686838] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8200.707572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8201.719391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8201.720805] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8201.753444] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8205.413566] Lustre: server umount lustre-OST0000 complete [ 8211.937260] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8211.951039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8211.961365] Lustre: Skipped 1 previous similar message [ 8211.985379] Lustre: Skipped 2 previous similar messages [ 8215.834645] Lustre: server umount lustre-MDT0000 complete [ 8217.069765] LustreError: 277739:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8217.094337] LustreError: 277739:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8219.697019] LustreError: 278511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447532 with bad export cookie 13410742961226927376 [ 8219.706699] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8220.012310] Lustre: server umount lustre-MDT0001 complete [ 8229.073239] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 8231.618229] Key type lgssc unregistered [ 8231.906986] LNet: 281453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8231.913934] LNetError: 281453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8231.927543] LNet: Removed LNI 192.168.203.134@tcp [ 8232.494100] Key type .llcrypt unregistered [ 8232.495779] Key type ._llcrypt unregistered [ 8243.031316] Key type ._llcrypt registered [ 8243.033654] Key type .llcrypt registered [ 8243.122335] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8260.058887] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 8274.176953] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 8274.853308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8274.904344] alg: No test for adler32 (adler32-zlib) [ 8275.984796] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8276.228845] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 8277.879190] Key type lgssc registered [ 8278.911270] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8308.748108] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8308.777143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8309.929716] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8309.978979] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8310.047606] Lustre: lustre-MDT0000: new disk, initializing [ 8310.138951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8310.159259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8313.594759] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8324.555821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8324.872692] Lustre: 285001:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8324.985760] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8324.990544] Lustre: Skipped 1 previous similar message [ 8325.151918] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8325.157613] Lustre: lustre-MDT0001: new disk, initializing [ 8325.375362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8325.444109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8325.463172] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8329.953852] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8340.146748] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8345.228325] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8350.752629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8350.955532] Lustre: lustre-OST0000: new disk, initializing [ 8350.958962] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8350.968441] Lustre: 286982:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8351.084601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8355.658616] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8357.406801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8357.413261] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8357.502704] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8363.831184] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8368.374435] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8368.583120] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8373.912748] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8374.272382] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8377.824324] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8377.829689] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8377.841194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8377.864660] Lustre: Skipped 1 previous similar message [ 8382.547553] Lustre: server umount lustre-OST0000 complete [ 8387.040471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8387.051132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8387.078332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8387.087383] Lustre: Skipped 1 previous similar message [ 8388.070362] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8393.073773] Lustre: server umount lustre-MDT0000 complete [ 8396.679079] LustreError: 284994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447709 with bad export cookie 1790640337899048284 [ 8396.688472] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8396.696640] LustreError: 284994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8397.003736] Lustre: server umount lustre-MDT0001 complete [ 8405.760694] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:01:56 (1788447716) [ 8412.226232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8412.667888] LustreError: 289392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8412.690445] LustreError: 289392:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8412.737307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8417.173090] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8417.767040] LustreError: 289393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8422.882311] LustreError: 289392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8425.152442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8425.545903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8429.717212] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8437.445351] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8443.031262] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8448.808925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8449.056191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8454.540442] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8461.703367] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8471.522186] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8471.535991] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8471.563767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8471.567805] Lustre: Skipped 2 previous similar messages [ 8473.589605] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8475.978829] Lustre: server umount lustre-OST0000 complete [ 8481.762069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8481.766167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8481.778988] Lustre: Skipped 1 previous similar message [ 8481.796803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8481.806210] Lustre: Skipped 2 previous similar messages [ 8486.367530] Lustre: server umount lustre-MDT0000 complete [ 8488.930294] LustreError: 289387:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8488.950411] LustreError: 289387:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8489.455788] LustreError: 289372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447802 with bad export cookie 1790640337899049166 [ 8489.462885] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8489.469954] LustreError: 289372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8489.716425] Lustre: server umount lustre-MDT0001 complete [ 8499.046285] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 8501.420387] Key type lgssc unregistered [ 8501.673583] LNet: 292927:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8501.684927] LNetError: 292927:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8501.709743] LNet: Removed LNI 192.168.203.134@tcp [ 8502.417154] Key type .llcrypt unregistered [ 8502.419360] Key type ._llcrypt unregistered [ 8524.998525] Key type ._llcrypt registered [ 8525.001430] Key type .llcrypt registered [ 8525.142981] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 8541.012436] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 8541.685530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8541.710301] alg: No test for adler32 (adler32-zlib) [ 8542.852260] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8543.041991] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 8544.703313] Key type lgssc registered [ 8545.716786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8588.684708] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8588.731770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8589.953825] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8589.975066] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8590.034500] Lustre: lustre-MDT0000: new disk, initializing [ 8590.107589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8590.128470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8593.721335] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8603.230081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8603.353114] Lustre: 296750:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8603.439620] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8603.444395] Lustre: Skipped 1 previous similar message [ 8603.526353] Lustre: lustre-MDT0001: new disk, initializing [ 8603.607449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8603.644269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8603.655240] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8608.272905] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8617.489805] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8622.787262] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8629.146507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8629.403214] Lustre: lustre-OST2710: new disk, initializing [ 8629.406512] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8629.411560] Lustre: 298721:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8629.500324] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8635.106015] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8638.529079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8638.546105] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8638.709664] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8645.396226] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8652.456929] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8652.630773] Lustre: lustre-OST03e8: new disk, initializing [ 8652.634798] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8652.643645] Lustre: 299783:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8652.751755] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8659.844686] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8662.168423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8662.200611] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8662.364514] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8667.264885] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8675.414544] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8675.617634] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8681.354142] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8681.627731] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8688.597852] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8689.040370] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8695.749881] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8696.139622] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8703.459198] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8703.463083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8703.470596] Lustre: Skipped 1 previous similar message [ 8708.582172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8708.599309] Lustre: Skipped 6 previous similar messages [ 8709.292750] Lustre: server umount lustre-MDT0000 complete [ 8712.970412] LustreError: 297546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448025 with bad export cookie 15807615223161476629 [ 8712.980341] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8713.340921] Lustre: server umount lustre-MDT0001 complete [ 8727.386294] Lustre: server umount lustre-OST2710 complete [ 8729.571428] Lustre: 294546:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788448026/real 1788448026] req@000000000067a042 x1875323498077056/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788448042 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8729.601231] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8729.608983] Lustre: Skipped 2 previous similar messages [ 8731.158897] Lustre: server umount lustre-OST03e8 complete [ 8745.869310] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 8753.321269] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 8781.906272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8782.291197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8782.329791] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8782.427800] Lustre: lustre-MDT0000: new disk, initializing [ 8782.490430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8782.518585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8786.582874] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8797.738916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8797.907252] Lustre: 305030:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8797.961635] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8797.965147] Lustre: Skipped 1 previous similar message [ 8798.014748] Lustre: lustre-MDT0001: new disk, initializing [ 8798.186694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8798.252121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8798.277384] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8803.653460] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8814.859854] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8823.180856] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8831.419408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8831.802757] Lustre: lustre-OST0000: new disk, initializing [ 8831.810569] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8831.828173] Lustre: 307000:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8831.935658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8837.193392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8837.224053] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8837.358250] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8837.839320] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8847.792535] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8852.621316] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8852.888961] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8857.094048] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8857.384211] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8863.211225] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8863.218724] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8863.225647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8863.235556] Lustre: Skipped 1 previous similar message [ 8865.215370] Lustre: server umount lustre-OST0000 complete [ 8869.856735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8869.876984] LustreError: Skipped 1 previous similar message [ 8869.909300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8869.926751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8869.940930] Lustre: Skipped 1 previous similar message [ 8875.774627] Lustre: server umount lustre-MDT0000 complete [ 8879.089365] LustreError: 305825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448191 with bad export cookie 15807615223161478323 [ 8879.096864] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8879.104846] LustreError: 305825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8879.398718] Lustre: server umount lustre-MDT0001 complete [ 8888.783810] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:09:58 (1788448198) [ 8891.053663] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8893.633437] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:10:03 (1788448203) [ 8898.043646] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 8925.424618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8925.761613] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8925.805744] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8925.969493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8930.739946] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8939.446883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8939.565430] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8939.614861] Lustre: 310859:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8945.237791] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8967.156311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8967.458819] Lustre: Denying initial registration attempt from nid 192.168.203.134@tcp, specified as failover [ 8967.470905] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8967.484824] LustreError: 312314:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -99 [ 8967.503638] LustreError: 312314:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 8967.518145] LustreError: 312314:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8967.611341] Lustre: server umount lustre-OST0000 complete [ 8967.614710] LustreError: 312314:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8975.843096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8975.854437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8975.866775] Lustre: Skipped 1 previous similar message [ 8975.878436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8975.881105] Lustre: Skipped 2 previous similar messages [ 8980.317271] Lustre: server umount lustre-MDT0000 complete [ 8981.986679] LustreError: 310866:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8982.022441] LustreError: 310866:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 8983.778080] LustreError: 310851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448296 with bad export cookie 15807615223161479205 [ 8983.782691] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8983.790578] LustreError: 310851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8984.051826] Lustre: server umount lustre-MDT0001 complete [ 8996.935904] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 9000.420706] Key type lgssc unregistered [ 9000.797778] LNet: 313434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9000.804394] LNetError: 313434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9000.831315] LNet: Removed LNI 192.168.203.134@tcp [ 9001.868744] Key type .llcrypt unregistered [ 9001.870793] Key type ._llcrypt unregistered [ 9012.370125] Key type ._llcrypt registered [ 9012.371640] Key type .llcrypt registered [ 9012.438946] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 9013.403702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9013.472302] alg: No test for adler32 (adler32-zlib) [ 9014.468654] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9014.807804] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 9016.503238] Key type lgssc registered [ 9017.722545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9067.080648] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 9075.908571] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9075.934442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9077.121993] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9077.134809] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9077.222991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9080.999171] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9090.341498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9090.484606] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9090.529276] Lustre: 316197:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9090.834666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9096.076700] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9103.370816] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9109.352666] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9129.201600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9129.489023] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9129.764180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9136.422052] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9145.012162] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9147.363038] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9147.365571] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9147.368286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9147.398156] Lustre: Skipped 1 previous similar message [ 9152.485436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9152.494119] Lustre: Skipped 1 previous similar message [ 9153.565274] Lustre: server umount lustre-OST0000 complete [ 9157.600845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9157.622344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9157.631585] Lustre: Skipped 2 previous similar messages [ 9162.724449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9162.743131] Lustre: Skipped 1 previous similar message [ 9163.591996] Lustre: server umount lustre-MDT0000 complete [ 9168.227099] LustreError: 316190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448480 with bad export cookie 5925734411938279359 [ 9168.248952] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9168.253668] LustreError: 316190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9168.887539] Lustre: server umount lustre-MDT0001 complete [ 9183.372377] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 9187.429218] Key type lgssc unregistered [ 9187.856853] LNet: 319770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9187.874245] LNetError: 319770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9187.933293] LNet: Removed LNI 192.168.203.134@tcp [ 9189.139938] Key type .llcrypt unregistered [ 9189.145212] Key type ._llcrypt unregistered [ 9214.817777] Key type ._llcrypt registered [ 9214.822873] Key type .llcrypt registered [ 9214.962934] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 9216.348627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9216.385147] alg: No test for adler32 (adler32-zlib) [ 9217.510351] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9217.874848] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 9219.579310] Key type lgssc registered [ 9221.085448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9229.296369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9229.321764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9230.870654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9235.521585] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9241.065674] LustreError: 321186:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9241.089975] LustreError: 321186:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9243.543580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9243.892317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9248.637227] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9256.487893] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9261.950568] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9270.891163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9271.423730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9277.561211] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9285.114459] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9299.424912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9299.429556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9299.438773] Lustre: Skipped 2 previous similar messages [ 9299.443430] Lustre: Skipped 1 previous similar message [ 9303.342956] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.34@tcp (stopping) [ 9303.345557] Lustre: Skipped 1 previous similar message [ 9304.553187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9305.502460] Lustre: server umount lustre-MDT0000 complete [ 9308.424990] LustreError: 321182:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9308.941181] LustreError: 321168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448621 with bad export cookie 5198617298766565106 [ 9308.953726] LustreError: 321168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9308.959729] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9309.666981] LustreError: 322983:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9309.671450] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9309.693883] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9309.701561] Lustre: Skipped 2 previous similar messages [ 9313.549147] LustreError: 321183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9313.698458] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9313.705726] Lustre: Skipped 1 previous similar message [ 9315.428699] Lustre: server umount lustre-MDT0001 complete [ 9321.280162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9329.274961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9334.434988] LustreError: 323019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9334.449453] LustreError: 323019:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9334.514078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9334.574130] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9338.745133] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9343.716037] LustreError: 324800:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9343.724570] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9343.732663] LustreError: 324800:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9346.462274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9346.898511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9351.191834] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9352.179431] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9358.228410] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9359.976857] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9364.936971] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9366.276825] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9372.650381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9372.654400] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9372.676723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9377.305289] Lustre: server umount lustre-OST0000 complete [ 9382.891217] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9382.893620] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9382.912293] Lustre: Skipped 1 previous similar message [ 9387.028635] Lustre: server umount lustre-MDT0000 complete [ 9388.001067] LustreError: 324805:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9388.025342] LustreError: 324805:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9391.042620] LustreError: 321167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448703 with bad export cookie 5198617298766572771 [ 9391.048316] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9391.053331] LustreError: 321167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9391.434779] Lustre: server umount lustre-MDT0001 complete [ 9401.324238] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 9403.754815] Key type lgssc unregistered [ 9403.958732] LNet: 327474:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9403.966831] LNetError: 327474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9403.994452] LNet: Removed LNI 192.168.203.134@tcp [ 9404.734056] Key type .llcrypt unregistered [ 9404.736352] Key type ._llcrypt unregistered [ 9430.512922] Key type ._llcrypt registered [ 9430.514525] Key type .llcrypt registered [ 9430.659679] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 9432.175680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9432.504467] alg: No test for adler32 (adler32-zlib) [ 9433.747689] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9434.188579] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 9435.991156] Key type lgssc registered [ 9437.375608] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9444.532562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9444.566636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9445.864904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9450.430434] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9456.099617] LustreError: 328888:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9456.115429] LustreError: 328888:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9458.588601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9458.978654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9464.291503] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9471.056944] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9475.422922] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9478.040515] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9478.053412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9478.059304] Lustre: Skipped 1 previous similar message [ 9483.232539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9483.236750] Lustre: Skipped 1 previous similar message [ 9484.259456] LustreError: 328889:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9484.269296] LustreError: 328889:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9484.300968] Lustre: server umount lustre-MDT0000 complete [ 9487.362537] LustreError: 328868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448800 with bad export cookie 11712950806374868421 [ 9487.372264] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9487.380507] LustreError: 328868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9487.678539] Lustre: server umount lustre-MDT0001 complete [ 9500.095447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9500.319342] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9500.354938] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9500.496795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9504.511587] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9512.719336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9512.818714] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9512.840331] Lustre: 331472:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9513.095336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9516.792501] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9523.130654] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9528.034302] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9536.872788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9537.145039] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9537.160849] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9537.170298] LustreError: 333359:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 9537.176441] LustreError: 333359:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9537.181464] LustreError: 333359:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9537.235272] Lustre: server umount lustre-OST0000 complete [ 9537.238822] LustreError: 333359:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9547.243555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9547.455477] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9547.643634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9553.104520] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9554.059034] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9558.840768] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9565.635667] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9565.736575] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9565.803632] Lustre: lustre-OST0001: new disk, initializing [ 9565.807516] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9565.814122] Lustre: 334902:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9565.909593] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9571.804231] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9574.974897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9574.995837] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9575.106640] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9578.486985] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9585.632354] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9585.641907] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9585.651702] LustreError: Skipped 1 previous similar message [ 9585.653397] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9585.673189] Lustre: Skipped 2 previous similar messages [ 9585.687339] Lustre: Skipped 1 previous similar message [ 9587.416333] Lustre: server umount lustre-OST0001 complete [ 9590.754894] LustreError: 333825:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9590.769480] LustreError: 333825:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9594.852418] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9594.861053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9594.881597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9595.890197] LustreError: 333825:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9595.913051] LustreError: 333825:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9597.157214] Lustre: server umount lustre-OST0000 complete [ 9600.996932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9601.018365] Lustre: Skipped 1 previous similar message [ 9601.031447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9601.047559] Lustre: Skipped 3 previous similar messages [ 9611.234500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9611.242095] Lustre: Skipped 3 previous similar messages [ 9615.328020] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9615.644522] Lustre: server umount lustre-MDT0000 complete [ 9616.355406] LustreError: 331479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9616.380748] LustreError: 331479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9619.406412] LustreError: 331466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448932 with bad export cookie 11712950806374868974 [ 9619.425236] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9619.945921] Lustre: server umount lustre-MDT0001 complete [ 9630.495198] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 9633.087268] Key type lgssc unregistered [ 9633.370255] LNet: 336859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9633.379612] LNetError: 336859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9633.394955] LNet: Removed LNI 192.168.203.134@tcp [ 9634.125392] Key type .llcrypt unregistered [ 9634.127708] Key type ._llcrypt unregistered [ 9706.497166] Key type ._llcrypt registered [ 9706.498980] Key type .llcrypt registered [ 9706.628789] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 9721.515179] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 9722.857894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9722.902914] alg: No test for adler32 (adler32-zlib) [ 9723.978936] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9724.244462] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 9725.929171] Key type lgssc registered [ 9727.219601] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9764.740330] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9764.763945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9766.105197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9766.172858] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9766.322626] Lustre: lustre-MDT0000: new disk, initializing [ 9766.471822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9766.518046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9771.704149] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9783.005702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9783.182916] Lustre: 341792:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9783.259120] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9783.263850] Lustre: Skipped 1 previous similar message [ 9783.377476] Lustre: lustre-MDT0001: new disk, initializing [ 9783.502784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9783.554696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9783.573113] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9788.307606] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9798.218528] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9804.318578] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9811.232199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9811.597761] Lustre: lustre-OST0000: new disk, initializing [ 9811.603589] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9811.612536] Lustre: 343761:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9811.736708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9817.801484] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9820.228049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9820.246180] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9820.346530] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9826.439953] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9830.937553] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9831.165979] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9836.026414] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9836.235767] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9841.127574] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9841.131717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9841.146142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9841.148769] Lustre: Skipped 1 previous similar message [ 9841.159724] Lustre: Skipped 1 previous similar message [ 9844.567492] Lustre: server umount lustre-OST0000 complete [ 9850.336410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9850.346741] LustreError: Skipped 1 previous similar message [ 9850.350844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9850.377311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9851.360537] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9854.480653] Lustre: server umount lustre-MDT0000 complete [ 9859.342599] LustreError: 341783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449172 with bad export cookie 16608294614003956662 [ 9859.349847] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9859.367196] LustreError: 341783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9859.774706] Lustre: server umount lustre-MDT0001 complete [ 9868.917384] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:26:19 (1788449179) [ 9875.655825] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:26:26 (1788449186) [ 9883.797076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9884.239938] LustreError: 346515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9884.260406] LustreError: 346515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9884.322674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9889.502296] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9889.762864] LustreError: 346516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9893.867143] LustreError: 346515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9898.723664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9898.977669] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9898.984201] Lustre: Skipped 2 previous similar messages [ 9899.175926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9904.012225] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9911.858179] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9917.819935] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9924.817550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9925.148487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9930.994178] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9938.728163] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9951.237691] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9953.024877] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9954.813355] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9956.598415] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9958.522599] Lustre: Failing over lustre-MDT0000 [ 9958.854763] Lustre: server umount lustre-MDT0000 complete [ 9958.886152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9958.895184] LustreError: 346511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9958.896750] Lustre: Skipped 2 previous similar messages [ 9958.928778] LustreError: 346511:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9964.014990] LustreError: 346512:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9964.051915] LustreError: 346512:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9972.501486] LustreError: 346509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9972.517322] LustreError: 346509:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9978.122700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9978.213777] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9978.484629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9978.553099] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9978.578354] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9983.000934] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9983.982198] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9984.027748] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 9984.077753] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9991.377198] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9992.680925] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9996.026649] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9997.962257] Lustre: Failing over lustre-MDT0000 [ 9998.219884] Lustre: server umount lustre-MDT0000 complete [ 9999.135962] LustreError: 349007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9999.159389] LustreError: 349007:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9999.333789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9999.352241] Lustre: Skipped 2 previous similar messages [ 9999.363644] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10015.695144] Lustre: 340016:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788449312/real 1788449312] req@00000000cd0ff73c x1875324736461056/t0(0) o400->MGC192.168.203.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788449328 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10015.714939] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10017.030205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10025.952817] LustreError: 340012:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005bf21453 x1875324736466560/t0(0) o250->MGC192.168.203.134@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10026.215855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10026.287084] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10029.849441] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10030.029554] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10031.594305] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10031.603331] Lustre: Skipped 2 previous similar messages [10031.655151] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10031.715993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10038.224284] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10039.837667] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10046.948399] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10046.950155] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10046.955287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10046.962495] Lustre: Skipped 1 previous similar message [10051.233875] Lustre: server umount lustre-MDT0000 complete [10052.069251] LustreError: 347244:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10052.085352] LustreError: 347244:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 28 previous similar messages [10054.724129] LustreError: 346496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449367 with bad export cookie 16608294614003959378 [10054.728290] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10054.736466] LustreError: 346496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10055.061584] Lustre: server umount lustre-MDT0001 complete [10069.043173] Lustre: server umount lustre-OST0000 complete [10084.319523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10090.579632] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10101.370654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10102.213246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10102.225973] Lustre: Skipped 1 previous similar message [10107.712458] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10116.563745] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10121.806372] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10128.389435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10129.776333] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10135.214930] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10144.552447] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10155.492200] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10155.504419] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10155.526993] Lustre: Skipped 1 previous similar message [10155.536422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10155.539673] Lustre: Skipped 2 previous similar messages [10160.811249] Lustre: server umount lustre-OST0000 complete [10167.778699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10167.793190] Lustre: Skipped 3 previous similar messages [10170.450704] Lustre: server umount lustre-MDT0000 complete [10172.898107] LustreError: 352880:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10172.912615] LustreError: 352880:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10173.624188] LustreError: 352860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449486 with bad export cookie 16608294614003960120 [10173.632770] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10173.639134] LustreError: 352860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10174.025661] Lustre: server umount lustre-MDT0001 complete [10185.829763] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [10189.045248] Key type lgssc unregistered [10189.390740] LNet: 356413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10189.407167] LNetError: 356413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10189.429970] LNet: Removed LNI 192.168.203.134@tcp [10190.459133] Key type .llcrypt unregistered [10190.462852] Key type ._llcrypt unregistered [10216.241594] Key type ._llcrypt registered [10216.244100] Key type .llcrypt registered [10216.384231] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [10217.549930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10217.569552] alg: No test for adler32 (adler32-zlib) [10218.663197] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10218.937533] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [10220.583534] Key type lgssc registered [10221.828177] Lustre: Echo OBD driver; http://www.lustre.org/ [10229.386144] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10229.422524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10230.797583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10234.999674] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10241.000765] LustreError: 357928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10241.029394] LustreError: 357928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10242.916024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10243.313813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10247.839616] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10255.376367] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10260.882530] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10267.190066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10267.563241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10273.590757] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10275.776509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10280.591849] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10291.177541] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10291.189265] Lustre: Skipped 1 previous similar message [10291.190392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10291.215346] Lustre: Skipped 2 previous similar messages [10292.699954] Lustre: server umount lustre-MDT0000 complete [10296.296070] LustreError: 357928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10296.325581] LustreError: 357928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10296.502159] LustreError: 357910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449609 with bad export cookie 13877980632025168438 [10296.506336] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10296.510622] LustreError: 357910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10297.155671] Lustre: server umount lustre-MDT0001 complete [10310.468761] Lustre: server umount lustre-OST0000 complete [10324.846845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10325.203083] LustreError: 361637:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10325.308661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10330.090556] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10330.603838] LustreError: 361638:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10334.694387] LustreError: 361637:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10338.778361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10339.090456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10343.089565] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10349.772676] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10354.935687] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10360.857526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10361.317419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10367.951983] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10369.540033] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10376.048872] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10389.988832] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10389.994765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10390.007860] Lustre: Skipped 3 previous similar messages [10390.012352] Lustre: Skipped 1 previous similar message [10394.820831] Lustre: server umount lustre-MDT0000 complete [10395.114781] LustreError: 362365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10395.143506] LustreError: 362365:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10398.488384] LustreError: 363481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449711 with bad export cookie 13877980632025169551 [10398.491357] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10398.518388] LustreError: 363481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10399.071877] Lustre: server umount lustre-MDT0001 complete [10413.462228] Lustre: server umount lustre-OST0000 complete [10426.833029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10427.226471] LustreError: 365547:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10427.320212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10432.722751] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10442.456310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10447.530521] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10455.097839] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10461.198371] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10468.390149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10468.757925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10468.764015] Lustre: Skipped 1 previous similar message [10471.863994] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10474.917467] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10482.250358] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10487.267335] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10487.287329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10489.217831] Lustre: server umount lustre-OST0000 complete [10493.412285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10493.418822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10493.426612] Lustre: Skipped 1 previous similar message [10493.448271] Lustre: Skipped 3 previous similar messages [10498.528810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10498.537115] Lustre: Skipped 1 previous similar message [10498.888143] Lustre: server umount lustre-MDT0000 complete [10502.502085] LustreError: 365528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449815 with bad export cookie 13877980632025170566 [10502.508895] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10502.522124] LustreError: 365528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10502.944621] Lustre: server umount lustre-MDT0001 complete [10514.582905] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [10517.919195] Key type lgssc unregistered [10518.327573] LNet: 369078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10518.336463] LNetError: 369078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10518.362946] LNet: Removed LNI 192.168.203.134@tcp [10519.137664] Key type .llcrypt unregistered [10519.144478] Key type ._llcrypt unregistered [10547.235728] Key type ._llcrypt registered [10547.237754] Key type .llcrypt registered [10547.320103] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [10548.137265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10548.239111] alg: No test for adler32 (adler32-zlib) [10549.303782] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10549.512656] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [10551.143177] Key type lgssc registered [10552.312681] Lustre: Echo OBD driver; http://www.lustre.org/ [10560.223631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10560.242045] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10560.254243] LustreError: 370519:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10560.290773] LustreError: 370519:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10560.295839] LustreError: 370519:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10560.301195] LustreError: 370519:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10560.307643] LustreError: 370519:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10567.997802] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10568.002704] LustreError: 370917:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10568.029890] LustreError: 370917:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10568.040213] LustreError: 370917:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10568.047943] LustreError: 370917:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10568.054305] LustreError: 370917:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10587.064131] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [10590.795586] Key type lgssc unregistered [10591.257364] LNet: 371919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10591.280050] LNetError: 371919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10591.316590] LNet: Removed LNI 192.168.203.134@tcp [10592.400677] Key type .llcrypt unregistered [10592.404978] Key type ._llcrypt unregistered [10611.505928] Key type ._llcrypt registered [10611.511467] Key type .llcrypt registered [10611.646838] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [10628.311391] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [10629.533296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10629.763064] alg: No test for adler32 (adler32-zlib) [10630.846315] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10631.143104] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [10632.856775] Key type lgssc registered [10634.246950] Lustre: Echo OBD driver; http://www.lustre.org/ [10669.763274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10669.824410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10671.032412] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10671.069305] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10671.143164] Lustre: lustre-MDT0000: new disk, initializing [10671.210117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10671.231706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10676.843740] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10688.463653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10688.593469] Lustre: 375074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10688.656622] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10688.660435] Lustre: Skipped 1 previous similar message [10688.732808] Lustre: lustre-MDT0001: new disk, initializing [10688.856291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10688.918528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10688.928338] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10693.423415] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10701.821844] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10707.074832] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10713.937800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10714.165635] Lustre: lustre-OST0000: new disk, initializing [10714.169125] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10714.173361] Lustre: 377041:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10714.262645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10720.227638] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10723.910095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10723.922929] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10724.048493] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10729.583292] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10734.157390] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10734.387560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10738.929425] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10739.170967] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10744.809656] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10744.811822] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10744.813698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10744.815933] LustreError: Skipped 1 previous similar message [10744.847945] Lustre: Skipped 1 previous similar message [10747.621047] Lustre: server umount lustre-OST0000 complete [10755.045130] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10755.069825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10755.082195] Lustre: Skipped 2 previous similar messages [10757.912563] Lustre: server umount lustre-MDT0000 complete [10761.779854] LustreError: 375064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450074 with bad export cookie 13758802413973941344 [10761.784561] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10761.801093] LustreError: 375064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10762.215803] Lustre: server umount lustre-MDT0001 complete [10771.659656] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10772.983981] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 11:41:23 (1788450083) [10779.160484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10779.622464] LustreError: 379502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10779.658255] LustreError: 379502:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10779.727357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10784.672830] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10784.742927] LustreError: 379503:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10789.859550] LustreError: 379502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10793.986251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10794.363622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10799.718222] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10807.036913] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10812.273565] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10818.825071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10819.286831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10825.725389] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10833.413496] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10841.042640] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10841.227661] Lustre: lustre-OST0001: new disk, initializing [10841.231844] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10841.240652] Lustre: 382339:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10841.333173] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10847.964331] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10848.876460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10848.894444] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10849.025250] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10857.585894] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10864.439555] Lustre: lustre-OST0001: Not available for connect from 192.168.203.34@tcp (stopping) [10864.610662] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10864.623241] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10864.645527] Lustre: Skipped 1 previous similar message [10867.838179] Lustre: server umount lustre-OST0001 complete [10869.519484] LustreError: 381323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10874.865742] LustreError: 381324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10874.890860] LustreError: 381324:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10885.092154] LustreError: 381323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10885.118755] LustreError: 381323:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10887.138515] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10887.146741] LustreError: Skipped 1 previous similar message [10887.151636] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10887.171549] Lustre: Skipped 1 previous similar message [10887.178153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10887.181173] Lustre: Skipped 2 previous similar messages [10892.756517] Lustre: server umount lustre-OST0000 complete [10896.865867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10896.873893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10896.903858] Lustre: Skipped 1 previous similar message [10896.911151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10896.922345] Lustre: Skipped 2 previous similar messages [10903.002196] Lustre: server umount lustre-MDT0000 complete [10905.570906] LustreError: 379502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10905.609732] LustreError: 379502:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10906.753963] LustreError: 382338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450219 with bad export cookie 13758802413973942226 [10906.755052] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10906.770217] LustreError: 382338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10913.327634] Lustre: server umount lustre-MDT0001 complete [10924.783218] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [10927.844375] Key type lgssc unregistered [10928.231136] LNet: 384323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10928.258876] LNetError: 384323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10928.286764] LNet: Removed LNI 192.168.203.134@tcp [10929.275283] Key type .llcrypt unregistered [10929.277437] Key type ._llcrypt unregistered [10980.425296] Key type ._llcrypt registered [10980.427708] Key type .llcrypt registered [10980.527599] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [10982.046727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10982.069854] alg: No test for adler32 (adler32-zlib) [10983.162641] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10983.414789] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [10985.103166] Key type lgssc registered [10986.147764] Lustre: Echo OBD driver; http://www.lustre.org/ [10993.467243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10993.485612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10994.673364] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10994.690576] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10994.855929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10999.117730] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11007.329987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11007.396494] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11007.446265] Lustre: 387120:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11007.701947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11011.458030] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11017.703870] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11022.786410] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11027.425374] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11027.429595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11027.449887] Lustre: Skipped 1 previous similar message [11027.459891] Lustre: Skipped 1 previous similar message [11030.700625] Lustre: server umount lustre-MDT0000 complete [11034.204342] LustreError: 387112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450346 with bad export cookie 13476404955203358830 [11034.223976] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11034.229271] LustreError: 387112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11034.579589] Lustre: server umount lustre-MDT0001 complete [11042.217482] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 11:45:52 (1788450352) [11047.513766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11047.885929] LustreError: 389628:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11047.910214] LustreError: 389628:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11047.934407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11051.663324] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11053.024246] LustreError: 389629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11058.150633] LustreError: 389629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11059.574828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11059.887819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11063.989532] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11071.137957] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11076.441600] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11082.618506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11082.905272] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11083.116476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11088.967511] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11095.358900] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11102.977382] Lustre: Permanently deactivating lustre-OST0000 [11103.001942] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11112.984502] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11114.602142] LustreError: 392664:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11114.612563] LustreError: 392664:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11114.617478] LustreError: 392664:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11118.021916] LustreError: 392712:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11118.025739] LustreError: 392712:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11126.053189] Lustre: server umount lustre-OST0000 complete [11128.465885] LustreError: 392961:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11128.473128] LustreError: 392961:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11131.872171] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11131.886374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11131.912537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11134.945308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11134.945361] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11140.064216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11140.073617] Lustre: Skipped 2 previous similar messages [11141.164652] Lustre: server umount lustre-MDT0000 complete [11144.882275] LustreError: 391461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450457 with bad export cookie 13476404955203359404 [11144.885406] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11144.896753] LustreError: 391461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11154.577624] Lustre: server umount lustre-MDT0001 complete [11162.276427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11166.475660] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11169.053176] LustreError: 394121:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11174.474909] Lustre: 394460:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.134@tcp [11176.133562] Lustre: server umount MGS complete [11186.258529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11186.739731] LustreError: 395185:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11186.770237] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11186.779121] Lustre: Skipped 1 previous similar message [11186.824259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11191.532525] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11192.289671] LustreError: 395186:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11199.127112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11199.382374] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11203.536345] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11210.807537] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11215.990857] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11217.515763] Lustre: Permanently reactivating lustre-OST0000 [11217.537420] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11217.545064] Lustre: Skipped 2 previous similar messages [11247.583180] LustreError: 397038:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11255.176600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11255.470806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11255.477862] Lustre: Skipped 1 previous similar message [11261.126694] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11268.587938] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11277.281736] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11277.288699] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11277.299452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11278.311411] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11279.533839] Lustre: server umount lustre-OST0000 complete [11286.496719] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11286.512601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11286.543514] Lustre: Skipped 1 previous similar message [11286.552515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11286.564434] Lustre: Skipped 2 previous similar messages [11289.838148] Lustre: server umount lustre-MDT0000 complete [11293.332499] LustreError: 395166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450606 with bad export cookie 13476404955203360461 [11293.333159] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11293.342052] LustreError: 395166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11293.767250] Lustre: server umount lustre-MDT0001 complete [11303.837821] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [11306.619405] Key type lgssc unregistered [11306.890230] LNet: 399690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11306.912949] LNetError: 399690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11306.948859] LNet: Removed LNI 192.168.203.134@tcp [11307.962946] Key type .llcrypt unregistered [11307.967474] Key type ._llcrypt unregistered [11327.445702] Key type ._llcrypt registered [11327.448510] Key type .llcrypt registered [11327.570488] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [11343.181988] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [11344.301670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11344.615915] alg: No test for adler32 (adler32-zlib) [11345.704073] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11346.108029] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [11347.855202] Key type lgssc registered [11349.421916] Lustre: Echo OBD driver; http://www.lustre.org/ [11384.662431] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11384.708303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11386.003246] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11386.077941] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11386.168083] Lustre: lustre-MDT0000: new disk, initializing [11386.236697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11386.251753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11390.617309] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11402.118628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11402.252216] Lustre: 402852:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11402.332992] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11402.344503] Lustre: Skipped 1 previous similar message [11402.414907] Lustre: lustre-MDT0001: new disk, initializing [11402.608243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11402.679133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11402.688019] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11407.886783] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11417.727220] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11423.471459] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11430.998355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11431.279357] Lustre: lustre-OST0000: new disk, initializing [11431.284037] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11431.293345] Lustre: 404819:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11431.404859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11437.374356] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11440.246264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11440.256784] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11440.457235] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11447.216618] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11452.242498] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11452.458926] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11456.313525] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11456.537193] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11461.092048] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11461.103697] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11461.104425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11461.112170] Lustre: Skipped 1 previous similar message [11464.603810] Lustre: server umount lustre-OST0000 complete [11469.283060] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11469.292279] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11469.313682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11469.329189] Lustre: Skipped 1 previous similar message [11471.330981] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11471.336840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11471.361462] Lustre: Skipped 1 previous similar message [11474.482837] Lustre: server umount lustre-MDT0000 complete [11477.889474] LustreError: 403646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450790 with bad export cookie 10000188809279963089 [11477.893149] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11477.902390] LustreError: 403646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11478.186537] Lustre: server umount lustre-MDT0001 complete [11485.283253] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11486.807646] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 11:53:17 (1788450797) [11492.679541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11493.044843] LustreError: 407276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11493.064040] LustreError: 407276:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11493.121211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11496.967822] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11498.465136] LustreError: 407277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11502.564368] LustreError: 407276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11503.921601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11504.272559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11507.837026] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11514.357348] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11519.396293] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11525.482663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11525.816473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11530.486097] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11536.675237] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11538.248593] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11540.576883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11548.650405] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11548.662772] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11548.664864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11548.672074] Lustre: Skipped 1 previous similar message [11551.861124] Lustre: server umount lustre-OST0000 complete [11555.297182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11555.309335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11555.326423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11555.332329] Lustre: Skipped 1 previous similar message [11561.134568] Lustre: server umount lustre-MDT0000 complete [11564.004903] LustreError: 407271:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11564.033652] LustreError: 407271:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11564.470173] LustreError: 407258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450877 with bad export cookie 10000188809279963971 [11564.477801] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11564.482128] LustreError: 407258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11564.842303] Lustre: server umount lustre-MDT0001 complete [11574.560980] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [11577.099723] Key type lgssc unregistered [11577.381634] LNet: 410998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11577.395419] LNetError: 410998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11578.472371] LNet: Removed LNI 192.168.203.134@tcp [11579.179231] Key type .llcrypt unregistered [11579.181563] Key type ._llcrypt unregistered [11598.641981] Key type ._llcrypt registered [11598.644830] Key type .llcrypt registered [11598.771865] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [11599.032394] Key type .llcrypt unregistered [11599.034669] Key type ._llcrypt unregistered [11620.478382] Key type ._llcrypt registered [11620.481994] Key type .llcrypt registered [11620.618550] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [11622.184301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11622.313148] alg: No test for adler32 (adler32-zlib) [11623.372418] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11623.580642] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [11625.271339] Key type lgssc registered [11626.352021] Lustre: Echo OBD driver; http://www.lustre.org/ [11633.310654] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11633.326949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11634.651905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11638.364834] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11644.900680] LustreError: 413203:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11644.934307] LustreError: 413203:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11646.190521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11646.524959] LustreError: 413202:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11646.557346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11651.563379] LustreError: 413203:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11651.587596] LustreError: 413203:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11652.091660] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11656.674775] LustreError: 413202:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11656.698411] LustreError: 413202:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11658.504625] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11659.759789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11664.205385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11664.537451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11667.956385] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11679.712648] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11679.728524] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11679.751157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11680.228704] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11680.229846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11680.252754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11680.259163] Lustre: Skipped 1 previous similar message [11681.990915] Lustre: server umount lustre-OST0000 complete [11689.956767] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11689.969712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11689.988318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11691.952081] Lustre: server umount lustre-MDT0000 complete [11695.591435] LustreError: 414644:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11695.615880] LustreError: 414644:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11695.681457] LustreError: 413184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451008 with bad export cookie 17689454961109514861 [11695.695662] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11696.007489] Lustre: server umount lustre-MDT0001 complete [11708.873249] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [11712.355446] Key type lgssc unregistered [11712.781533] LNet: 416349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11712.806194] LNetError: 416349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11713.839926] LNet: Removed LNI 192.168.203.134@tcp [11714.690850] Key type .llcrypt unregistered [11714.701226] Key type ._llcrypt unregistered [11737.930128] Key type ._llcrypt registered [11737.931682] Key type .llcrypt registered [11738.047642] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [11738.783741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11738.881035] alg: No test for adler32 (adler32-zlib) [11739.872259] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11740.128366] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [11741.760182] Key type lgssc registered [11742.782328] Lustre: Echo OBD driver; http://www.lustre.org/ [11750.864824] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11750.880597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11766.433981] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11771.551303] LustreError: 417737:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.134@tcp: failed processing log, type 4: rc = -110 [11798.175187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11804.202192] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11810.901938] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11817.007544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11817.552248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11817.617716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11821.824763] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11822.561644] LustreError: 417763:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11822.589857] LustreError: 417763:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11827.684380] LustreError: 417762:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11829.332623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11829.740295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11833.651398] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11840.842923] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11846.430869] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11855.329335] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11855.346638] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11855.363227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11855.825766] Lustre: server umount lustre-OST0000 complete [11860.448774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11860.465375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11860.496348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11863.524090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11863.526388] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11868.646673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11868.660186] Lustre: Skipped 2 previous similar messages [11869.747411] Lustre: server umount lustre-MDT0000 complete [11873.345108] LustreError: 417743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451186 with bad export cookie 322329127198932296 [11873.349394] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11873.359731] LustreError: 417743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11873.761640] Lustre: server umount lustre-MDT0001 complete [11884.595180] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [11887.275840] Key type lgssc unregistered [11887.603629] LNet: 421303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11887.619570] LNetError: 421303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11887.636819] LNet: Removed LNI 192.168.203.134@tcp [11888.511958] Key type .llcrypt unregistered [11888.519476] Key type ._llcrypt unregistered [11913.448289] Key type ._llcrypt registered [11913.451032] Key type .llcrypt registered [11913.538220] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [11914.510769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11914.573166] alg: No test for adler32 (adler32-zlib) [11915.683100] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11915.891223] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [11917.559156] Key type lgssc registered [11918.881767] Lustre: Echo OBD driver; http://www.lustre.org/ [11926.932471] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11926.944925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11928.393557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11932.921271] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11938.794558] LustreError: 422714:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11938.807781] LustreError: 422714:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11941.000162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11941.397055] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11945.331593] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11952.055767] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11957.456901] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11963.817274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11964.215063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11969.928593] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11973.431725] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11976.528785] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11982.307502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11982.320274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11982.342216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11983.840504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11983.842589] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11987.065790] Lustre: server umount lustre-MDT0000 complete [11988.962171] LustreError: 422711:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11988.975349] LustreError: 422711:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11989.156428] LustreError: 422694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.34@tcp arrived at 1788451301 with bad export cookie 2656818217759377327 [11990.726224] LustreError: 422696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451303 with bad export cookie 2656818217759376431 [11990.745727] LustreError: 422696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11990.751289] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11993.055961] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11993.061377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11993.069918] Lustre: Skipped 1 previous similar message [11993.074222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11993.078183] Lustre: Skipped 3 previous similar messages [11994.080878] LustreError: 422711:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11994.106113] LustreError: 422711:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11995.107326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11995.114097] Lustre: Skipped 1 previous similar message [11996.973477] Lustre: server umount lustre-OST0000 complete [12000.227111] LustreError: 422714:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12000.244715] LustreError: 422714:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12003.775540] Lustre: server umount lustre-MDT0001 complete [12015.067949] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [12018.019151] Key type lgssc unregistered [12018.308214] LNet: 426409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12018.325672] LNetError: 426409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12019.370959] LNet: Removed LNI 192.168.203.134@tcp [12020.168558] Key type .llcrypt unregistered [12020.171736] Key type ._llcrypt unregistered [12044.243804] Key type ._llcrypt registered [12044.248272] Key type .llcrypt registered [12044.373212] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [12045.232577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12045.251646] alg: No test for adler32 (adler32-zlib) [12046.267559] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12046.540044] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [12048.231279] Key type lgssc registered [12049.472299] Lustre: Echo OBD driver; http://www.lustre.org/ [12057.040287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12057.060643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12058.495062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12063.388684] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [12068.846391] LustreError: 427822:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12068.861597] LustreError: 427822:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12071.596418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12072.000886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12076.502435] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [12083.780619] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12089.560199] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12096.438693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12096.747634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12103.283697] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [12103.936427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12111.134613] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12117.984094] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12117.998763] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12118.018476] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12118.798825] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.34@tcp (stopping) [12119.524377] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12121.996524] Lustre: server umount lustre-MDT0001 complete [12123.951218] LustreError: 427818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12129.767178] LustreError: 427819:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12129.795244] LustreError: 427819:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12134.884889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12134.899194] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12134.899525] LustreError: 427822:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12134.922227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12134.961426] Lustre: Skipped 2 previous similar messages [12136.386854] Lustre: server umount lustre-OST0000 complete [12140.000254] LustreError: 427823:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12140.811256] Lustre: server umount lustre-MDT0000 complete [12154.004220] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [12157.813185] Key type lgssc unregistered [12158.125564] LNet: 431508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12158.138704] LNetError: 431508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12158.173801] LNet: Removed LNI 192.168.203.134@tcp [12159.544394] Key type .llcrypt unregistered [12159.551483] Key type ._llcrypt unregistered [12186.182353] Key type ._llcrypt registered [12186.185505] Key type .llcrypt registered [12186.356944] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [12201.004446] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [12201.961234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12202.043894] alg: No test for adler32 (adler32-zlib) [12203.058779] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12203.316662] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [12204.975203] Key type lgssc registered [12206.160142] Lustre: Echo OBD driver; http://www.lustre.org/ [12242.643163] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [12257.270352] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12257.315729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12258.584446] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12258.626325] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12258.736553] Lustre: lustre-MDT0000: new disk, initializing [12258.805769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12258.819248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12263.003744] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [12277.835780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12278.031568] Lustre: lustre-OST0000: new disk, initializing [12278.034981] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12278.038397] Lustre: Skipped 1 previous similar message [12278.042309] Lustre: 436535:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12278.093766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12284.275451] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [12285.513873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12285.525922] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12285.559344] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12293.537689] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12307.971737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12308.156337] Lustre: 435517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12308.249222] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12308.351779] Lustre: lustre-MDT0001: new disk, initializing [12308.528135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12308.583518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12308.588265] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12313.410245] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [12323.019862] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12323.253811] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12327.633746] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12327.963450] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12333.028084] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12333.029372] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12333.058995] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12333.070370] Lustre: Skipped 1 previous similar message [12335.922725] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.34@tcp (stopping) [12338.153465] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12338.167213] Lustre: Skipped 1 previous similar message [12341.001779] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.34@tcp (stopping) [12346.128614] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.34@tcp (stopping) [12346.134893] Lustre: Skipped 2 previous similar messages [12347.359374] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12347.545983] Lustre: server umount lustre-MDT0001 complete [12353.505042] LustreError: 435525:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12353.528107] LustreError: 435525:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12358.629596] LustreError: 435529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12358.645991] LustreError: 435529:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12363.745994] LustreError: 435526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12363.773323] LustreError: 435526:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12367.330025] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12367.336170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12367.345028] Lustre: Skipped 1 previous similar message [12367.348916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12368.876496] LustreError: 435530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12371.358416] Lustre: server umount lustre-OST0000 complete [12373.989167] LustreError: 435529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12389.343125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12389.739658] Lustre: server umount lustre-MDT0000 complete [12403.870139] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [12406.414693] Key type lgssc unregistered [12406.845836] LNet: 440287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12406.864386] LNetError: 440287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12406.897452] LNet: Removed LNI 192.168.203.134@tcp [12407.912155] Key type .llcrypt unregistered [12407.917195] Key type ._llcrypt unregistered [12448.514815] Key type ._llcrypt registered [12448.520124] Key type .llcrypt registered [12448.640348] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [12448.903392] Key type .llcrypt unregistered [12448.906770] Key type ._llcrypt unregistered [12465.227095] Key type ._llcrypt registered [12465.228757] Key type .llcrypt registered [12465.336380] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [12466.019553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12466.145120] alg: No test for adler32 (adler32-zlib) [12467.151564] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12467.315248] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [12468.960405] Key type lgssc registered [12470.426947] Lustre: Echo OBD driver; http://www.lustre.org/ [12497.525417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12497.581994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12498.876325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12498.903719] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12499.046675] Lustre: lustre-MDT0000: new disk, initializing [12499.136361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12499.153479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12503.538128] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [12513.703934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12513.823240] Lustre: 443654:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12513.865316] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12513.868725] Lustre: Skipped 1 previous similar message [12513.928659] Lustre: lustre-MDT0001: new disk, initializing [12514.022229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12514.058104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12514.072279] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12518.223477] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [12527.366191] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12533.368294] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12540.618384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12541.017534] Lustre: lustre-OST0000: new disk, initializing [12541.028098] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12541.035803] Lustre: 445622:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12541.133500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12546.606614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12546.620313] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12546.742876] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12547.850362] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [12557.806490] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12565.482367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12565.496424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12565.516318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12567.524367] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12567.534304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12567.547130] Lustre: Skipped 1 previous similar message [12567.572509] Lustre: Skipped 2 previous similar messages [12572.180864] Lustre: server umount lustre-MDT0000 complete [12575.266080] LustreError: 443646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451888 with bad export cookie 8610174693351231394 [12575.276248] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12575.288806] LustreError: 443646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12575.591125] Lustre: server umount lustre-MDT0001 complete [12588.569380] Lustre: server umount lustre-OST0000 complete [12598.699244] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:11:49 (1788451909) [12606.585875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12607.126702] LustreError: 447648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12607.179937] LustreError: 447648:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12607.198340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12610.945164] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [12612.576158] LustreError: 447649:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12616.676326] LustreError: 447648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12621.178652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12621.554969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12625.670909] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [12634.566158] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12640.751880] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12660.264483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12660.467474] Lustre: Found index 0 for lustre-OST0000, updating log [12660.715401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12666.454097] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [12673.310926] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12678.126302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12688.352924] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12688.355140] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12688.360967] LustreError: Skipped 1 previous similar message [12688.363980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12688.390660] Lustre: Skipped 1 previous similar message [12689.621213] Lustre: server umount lustre-OST0000 complete [12705.610204] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12705.641574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12705.650741] Lustre: Skipped 4 previous similar messages [12710.880403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12710.890540] Lustre: Skipped 1 previous similar message [12712.146185] Lustre: server umount lustre-MDT0000 complete [12715.953710] LustreError: 447630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452028 with bad export cookie 8610174693351232787 [12715.967051] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12715.975644] LustreError: 447630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12716.354895] Lustre: server umount lustre-MDT0001 complete [12717.280095] LNet: 1 local NIs in recovery (showing 1): 192.168.203.134@tcp [12724.970316] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:13:55 (1788452035) [13345.084550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13345.671481] LustreError: 454058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13345.765198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13350.438753] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [13350.880277] LustreError: 454059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13356.000799] LustreError: 454058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13359.380047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13359.907845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13364.300837] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [13371.563386] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13377.463662] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13395.172807] LNet: 1 local NIs in recovery (showing 1): 192.168.203.134@tcp [13397.431095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13397.734659] Lustre: Found index 0 for lustre-OST0000, updating log [13398.029291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13404.029855] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [13411.365779] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13413.396869] Lustre: server umount lustre-OST0000 complete [13418.024330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13418.041701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13418.044261] Lustre: Skipped 2 previous similar messages [13419.104861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13419.111965] Lustre: Skipped 2 previous similar messages [13421.156077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13421.163116] Lustre: Skipped 3 previous similar messages [13425.262145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13425.274495] Lustre: Skipped 7 previous similar messages [13431.263128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13431.396663] LustreError: 455080:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13431.408042] LustreError: 455080:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13431.478932] Lustre: server umount lustre-MDT0000 complete [13434.996697] LustreError: 454039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452747 with bad export cookie 8610174693351233872 [13435.019314] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13435.333220] Lustre: server umount lustre-MDT0001 complete [13454.559242] LNet: 1 local NIs in recovery (showing 1): 192.168.203.134@tcp [13465.578951] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [13477.306115] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [13513.012916] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:27:03 (1788452823) [13531.418687] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [13534.354681] Key type lgssc unregistered [13534.647411] LNet: 460669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13534.659586] LNetError: 460669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13657.573193] LNet: Removed LNI 192.168.203.134@tcp [13658.580828] Key type .llcrypt unregistered [13658.588179] Key type ._llcrypt unregistered [13675.184346] Key type ._llcrypt registered [13675.189200] Key type .llcrypt registered [13675.297984] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [13676.794072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13676.816424] alg: No test for adler32 (adler32-zlib) [13677.974494] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13678.325656] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [13680.007418] Key type lgssc registered [13681.225327] Lustre: Echo OBD driver; http://www.lustre.org/ [13685.530725] LNet: Added LNI 192.168.203.134@tcp100 [8/256/0/180] [13686.415683] LNet: Added LNI 192.168.203.134@tcp101 [8/256/0/180] [13687.287196] LNet: Added LNI 192.168.203.134@tcp102 [8/256/0/180] [13688.123871] LNet: Added LNI 192.168.203.134@tcp103 [8/256/0/180] [13689.035210] LNet: Added LNI 192.168.203.134@tcp104 [8/256/0/180] [13689.907330] LNet: Added LNI 192.168.203.134@tcp105 [8/256/0/180] [13690.702602] LNet: Added LNI 192.168.203.134@tcp106 [8/256/0/180] [13691.572559] LNet: Added LNI 192.168.203.134@tcp107 [8/256/0/180] [13692.563153] LNet: Added LNI 192.168.203.134@tcp108 [8/256/0/180] [13693.414992] LNet: Added LNI 192.168.203.134@tcp109 [8/256/0/180] [13694.401587] LNet: Added LNI 192.168.203.134@tcp110 [8/256/0/180] [13695.466498] LNet: Added LNI 192.168.203.134@tcp111 [8/256/0/180] [13696.392782] LNet: Added LNI 192.168.203.134@tcp112 [8/256/0/180] [13697.272517] LNet: Added LNI 192.168.203.134@tcp113 [8/256/0/180] [13698.169932] LNet: Added LNI 192.168.203.134@tcp114 [8/256/0/180] [13698.976657] LNet: Added LNI 192.168.203.134@tcp115 [8/256/0/180] [13699.719445] LNet: Added LNI 192.168.203.134@tcp116 [8/256/0/180] [13700.452518] LNet: Added LNI 192.168.203.134@tcp117 [8/256/0/180] [13701.278402] LNet: Added LNI 192.168.203.134@tcp118 [8/256/0/180] [13702.234971] LNet: Added LNI 192.168.203.134@tcp119 [8/256/0/180] [13703.413605] LNet: Added LNI 192.168.203.134@tcp120 [8/256/0/180] [13704.412860] LNet: Added LNI 192.168.203.134@tcp121 [8/256/0/180] [13705.248212] LNet: Added LNI 192.168.203.134@tcp122 [8/256/0/180] [13706.112512] LNet: Added LNI 192.168.203.134@tcp123 [8/256/0/180] [13707.235134] LNet: Added LNI 192.168.203.134@tcp124 [8/256/0/180] [13708.264836] LNet: Added LNI 192.168.203.134@tcp125 [8/256/0/180] [13709.177141] LNet: Added LNI 192.168.203.134@tcp126 [8/256/0/180] [13709.932614] LNet: Added LNI 192.168.203.134@tcp127 [8/256/0/180] [13710.922241] LNet: Added LNI 192.168.203.134@tcp128 [8/256/0/180] [13712.088160] LNet: Added LNI 192.168.203.134@tcp129 [8/256/0/180] [13713.459633] LNet: Added LNI 192.168.203.134@tcp130 [8/256/0/180] [13714.518588] LNet: Added LNI 192.168.203.134@tcp131 [8/256/0/180] [13715.480236] LNet: Added LNI 192.168.203.134@tcp132 [8/256/0/180] [13716.487763] LNet: Added LNI 192.168.203.134@tcp133 [8/256/0/180] [13717.569121] LNet: Added LNI 192.168.203.134@tcp134 [8/256/0/180] [13718.661565] LNet: Added LNI 192.168.203.134@tcp135 [8/256/0/180] [13727.020984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13727.051305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13728.123742] Lustre: 463700:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.134@tcp: can't add NID 192.168.203.134@tcp130: rc = -75 [13728.245678] LNetError: 461158:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.134 [13728.257250] LNetError: Protocol error connecting to 192.168.203.134@tcp120 at host 192.168.203.134:988: is it running a compatible version of Lustre? [13728.279214] LNetError: Skipped 3 previous similar messages [13728.370302] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13728.426366] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13728.542503] Lustre: lustre-MDT0000: new disk, initializing [13728.543278] LNet: 6 local NIs in recovery (showing 6): 192.168.203.134@tcp123, 192.168.203.134@tcp125, 192.168.203.134@tcp120, 192.168.203.134@tcp128, 192.168.203.134@tcp127, 192.168.203.134@tcp129 [13728.622978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13728.644380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13729.313278] LNetError: 463730:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.134 [13729.323171] LNetError: 463730:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [13729.334727] LNetError: Protocol error connecting to 192.168.203.134@tcp100 at host 192.168.203.134:988: is it running a compatible version of Lustre? [13729.346233] LNetError: Skipped 6 previous similar messages [13731.617641] LNetError: 463730:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.134 [13731.634442] LNetError: 463730:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [13731.654885] LNetError: Protocol error connecting to 192.168.203.134@tcp125 at host 192.168.203.134:988: is it running a compatible version of Lustre? [13731.681812] LNetError: Skipped 10 previous similar messages [13734.384348] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [13735.726841] LNetError: 461160:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.134 [13735.748259] LNetError: Protocol error connecting to 192.168.203.134@tcp125 at host 192.168.203.134:988: is it running a compatible version of Lustre? [13743.904130] LNetError: 461159:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.134 [13743.923683] LNetError: Protocol error connecting to 192.168.203.134@tcp125 at host 192.168.203.134:988: is it running a compatible version of Lustre? [13746.623437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13746.909306] Lustre: 463718:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13746.999580] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13747.009384] Lustre: Skipped 1 previous similar message [13747.163747] Lustre: lustre-MDT0001: new disk, initializing [13747.234547] Lustre: 464518:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.134@tcp: can't add NID 192.168.203.134@tcp130: rc = -75 [13747.255323] Lustre: 464518:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13747.456725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13747.515338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13747.532757] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13753.966712] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [13754.410454] Lustre: 464798:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.134@tcp: can't add NID 192.168.203.134@tcp130: rc = -75 [13754.415993] Lustre: 464798:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13758.220252] LNetError: 461157:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.134 [13758.240060] LNetError: Protocol error connecting to 192.168.203.134@tcp115 at host 192.168.203.134:988: is it running a compatible version of Lustre? [13765.608694] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13773.502352] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13774.560248] LNetError: 461160:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.134 [13774.600024] LNetError: 461160:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 13 previous similar messages [13774.619305] LNetError: Protocol error connecting to 192.168.203.134@tcp100 at host 192.168.203.134:988: is it running a compatible version of Lustre? [13774.644627] LNetError: Skipped 13 previous similar messages [13782.068586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13782.567411] Lustre: lustre-OST0000: new disk, initializing [13782.572975] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13782.581102] Lustre: 465694:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13782.736299] Lustre: 465699:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.134@tcp: can't add NID 192.168.203.134@tcp130: rc = -75 [13782.747305] Lustre: 465699:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13782.861620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13788.960146] LNet: 1 local NIs in recovery (showing 1): 192.168.203.134@tcp125 [13790.360575] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [13790.793458] Lustre: 466065:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.134@tcp: can't add NID 192.168.203.134@tcp130: rc = -75 [13790.808518] Lustre: 466065:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13790.889771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13790.899221] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13791.116534] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13801.366323] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13809.379112] LNetError: 461159:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.134 [13809.395790] LNetError: 461159:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 21 previous similar messages [13809.401756] LNetError: Protocol error connecting to 192.168.203.134@tcp115 at host 192.168.203.134:988: is it running a compatible version of Lustre? [13809.417425] LNetError: Skipped 21 previous similar messages [13811.685421] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13811.685594] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13811.716234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13811.753197] Lustre: Skipped 1 previous similar message [13815.739720] Lustre: server umount lustre-OST0000 complete [13821.924060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13821.925996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13821.939292] Lustre: Skipped 2 previous similar messages [13821.953883] Lustre: Skipped 1 previous similar message [13826.610434] Lustre: server umount lustre-MDT0000 complete [13830.780949] LustreError: 464517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453143 with bad export cookie 8975700578790005602 [13830.786872] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13830.795799] LustreError: 464517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13831.292693] Lustre: server umount lustre-MDT0001 complete [13842.019904] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [13845.090807] Key type lgssc unregistered [13845.583945] LNet: 467606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13845.601258] LNetError: 467606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13845.607930] LNet: Removed LNI 192.168.203.134@tcp [13846.623174] LNet: Removed LNI 192.168.203.134@tcp100 [13846.625280] LNetError: 467606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13846.647566] LNetError: 467606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13846.657310] LNet: Removed LNI 192.168.203.134@tcp101 [13847.713570] LNet: Removed LNI 192.168.203.134@tcp102 [13847.725285] LNetError: 467606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13847.749400] LNetError: 467606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13847.772610] LNet: Removed LNI 192.168.203.134@tcp103 [13848.799209] LNet: Removed LNI 192.168.203.134@tcp104 [13849.823138] LNet: Removed LNI 192.168.203.134@tcp105 [13849.825369] LNetError: 467606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13849.838307] LNetError: 467606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13849.852082] LNet: Removed LNI 192.168.203.134@tcp106 [13850.911384] LNet: Removed LNI 192.168.203.134@tcp107 [13851.947110] LNet: Removed LNI 192.168.203.134@tcp108 [13853.024970] LNet: Removed LNI 192.168.203.134@tcp109 [13854.047300] LNet: Removed LNI 192.168.203.134@tcp110 [13854.050154] LNetError: 467606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13854.061949] LNetError: 467606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13854.072071] LNet: Removed LNI 192.168.203.134@tcp111 [13855.135253] LNet: Removed LNI 192.168.203.134@tcp112 [13856.159177] LNet: Removed LNI 192.168.203.134@tcp113 [13857.185468] LNet: Removed LNI 192.168.203.134@tcp114 [13858.207190] LNet: Removed LNI 192.168.203.134@tcp115 [13859.231332] LNet: Removed LNI 192.168.203.134@tcp116 [13860.259783] LNet: Removed LNI 192.168.203.134@tcp117 [13861.280059] LNet: Removed LNI 192.168.203.134@tcp118 [13862.303141] LNet: Removed LNI 192.168.203.134@tcp119 [13862.307570] LNetError: 467606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13862.315294] LNetError: 467606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13862.333296] LNet: Removed LNI 192.168.203.134@tcp120 [13863.391826] LNet: Removed LNI 192.168.203.134@tcp121 [13864.415216] LNet: Removed LNI 192.168.203.134@tcp122 [13865.439219] LNet: Removed LNI 192.168.203.134@tcp123 [13866.463242] LNet: Removed LNI 192.168.203.134@tcp124 [13866.474642] LNet: Removed LNI 192.168.203.134@tcp125 [13867.489047] LNet: Removed LNI 192.168.203.134@tcp126 [13868.512717] LNet: Removed LNI 192.168.203.134@tcp127 [13869.535198] LNet: Removed LNI 192.168.203.134@tcp128 [13870.562918] LNet: Removed LNI 192.168.203.134@tcp129 [13870.565154] LNet: Removed LNI 192.168.203.134@tcp130 [13870.570667] LNet: Removed LNI 192.168.203.134@tcp131 [13870.574691] LNet: Removed LNI 192.168.203.134@tcp132 [13870.579730] LNet: Removed LNI 192.168.203.134@tcp133 [13870.590287] LNet: Removed LNI 192.168.203.134@tcp134 [13870.601933] LNet: Removed LNI 192.168.203.134@tcp135 [13871.361389] Key type .llcrypt unregistered [13871.366740] Key type ._llcrypt unregistered [13901.227438] Key type ._llcrypt registered [13901.229787] Key type .llcrypt registered [13901.392716] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [13917.884944] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [13919.172059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13919.239530] alg: No test for adler32 (adler32-zlib) [13920.573262] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13920.949464] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [13922.719177] Key type lgssc registered [13924.320985] Lustre: Echo OBD driver; http://www.lustre.org/ [13960.373345] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 12:34:30 (1788453270) [13977.935564] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [13980.646779] Key type lgssc unregistered [13980.968031] LNet: 471688:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13980.980838] LNetError: 471688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13981.008854] LNet: Removed LNI 192.168.203.134@tcp [13981.943656] Key type .llcrypt unregistered [13981.947126] Key type ._llcrypt unregistered [13999.891123] Key type ._llcrypt registered [13999.892822] Key type .llcrypt registered [13999.995253] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [14000.917107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14001.105507] alg: No test for adler32 (adler32-zlib) [14002.128759] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [14002.328823] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [14004.015283] Key type lgssc registered [14005.164644] Lustre: Echo OBD driver; http://www.lustre.org/ [14014.371583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14014.407747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14015.677112] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14015.699634] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14015.771256] Lustre: lustre-MDT0000: new disk, initializing [14015.886329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14015.915605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14020.586479] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [14032.580707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14032.853811] Lustre: 472899:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14032.987330] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14032.995094] Lustre: Skipped 1 previous similar message [14033.186486] Lustre: lustre-MDT0001: new disk, initializing [14033.344547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14033.485554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14033.529454] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14040.213429] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [14050.539304] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14055.886969] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14062.473799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14062.880870] Lustre: lustre-OST0000: new disk, initializing [14062.885125] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14062.893507] Lustre: 474868:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14063.059977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14069.402689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14069.420896] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14069.628835] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14070.280662] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [14082.593360] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14086.799894] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing lnet_if_list [14094.223524] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14094.277989] LNet: Added LNI 192.168.203.134@tcp10 [8/256/0/180] [14100.812961] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14100.832376] LNet: Added LNI 192.168.203.134@tcp11 [8/256/0/180] [14105.440981] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14105.499599] LNet: Added LNI 192.168.203.134@tcp12 [8/256/0/180] [14110.391459] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14110.416139] LNet: Added LNI 192.168.203.134@tcp13 [8/256/0/180] [14116.051221] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14116.120604] LNet: Added LNI 192.168.203.134@tcp14 [8/256/0/180] [14121.719559] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14121.798435] LNet: Added LNI 192.168.203.134@tcp15 [8/256/0/180] [14126.267259] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14126.302172] LNet: Added LNI 192.168.203.134@tcp16 [8/256/0/180] [14133.058425] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14133.094970] LNet: Added LNI 192.168.203.134@tcp17 [8/256/0/180] [14138.187852] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14138.214658] LNet: Added LNI 192.168.203.134@tcp18 [8/256/0/180] [14143.508650] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14143.581960] LNet: Added LNI 192.168.203.134@tcp19 [8/256/0/180] [14148.276438] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14148.299383] LNet: Added LNI 192.168.203.134@tcp20 [8/256/0/180] [14152.886201] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14152.941620] LNet: Added LNI 192.168.203.134@tcp21 [8/256/0/180] [14158.522042] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14158.545157] LNet: Added LNI 192.168.203.134@tcp22 [8/256/0/180] [14163.775280] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14163.802628] LNet: Added LNI 192.168.203.134@tcp23 [8/256/0/180] [14170.701502] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14170.729477] LNet: Added LNI 192.168.203.134@tcp24 [8/256/0/180] [14176.648595] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14176.675114] LNet: Added LNI 192.168.203.134@tcp25 [8/256/0/180] [14182.588907] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14182.620723] LNet: Added LNI 192.168.203.134@tcp26 [8/256/0/180] [14188.308326] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14188.353608] LNet: Added LNI 192.168.203.134@tcp27 [8/256/0/180] [14195.075673] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14195.157397] LNet: Added LNI 192.168.203.134@tcp28 [8/256/0/180] [14203.571688] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14203.611549] LNet: Added LNI 192.168.203.134@tcp29 [8/256/0/180] [14211.721114] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14211.770417] LNet: Added LNI 192.168.203.134@tcp30 [8/256/0/180] [14218.624446] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14218.672434] LNet: Added LNI 192.168.203.134@tcp31 [8/256/0/180] [14226.341528] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14226.391426] LNet: Added LNI 192.168.203.134@tcp32 [8/256/0/180] [14233.701331] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14233.788417] LNet: Added LNI 192.168.203.134@tcp33 [8/256/0/180] [14241.236408] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14241.282141] LNet: Added LNI 192.168.203.134@tcp34 [8/256/0/180] [14248.045495] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14248.117788] LNet: Added LNI 192.168.203.134@tcp35 [8/256/0/180] [14254.915363] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14254.945645] LNet: Added LNI 192.168.203.134@tcp36 [8/256/0/180] [14263.096785] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14263.144898] LNet: Added LNI 192.168.203.134@tcp37 [8/256/0/180] [14271.243967] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14271.274821] LNet: Added LNI 192.168.203.134@tcp38 [8/256/0/180] [14279.862383] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14279.892995] LNet: Added LNI 192.168.203.134@tcp39 [8/256/0/180] [14287.826526] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14287.868599] LNet: Added LNI 192.168.203.134@tcp40 [8/256/0/180] [14295.867548] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14295.890471] LNet: Added LNI 192.168.203.134@tcp41 [8/256/0/180] [14303.521953] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14303.585641] LNet: Added LNI 192.168.203.134@tcp42 [8/256/0/180] [14311.278415] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14311.298851] LNet: Added LNI 192.168.203.134@tcp43 [8/256/0/180] [14319.263190] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14319.307832] LNet: Added LNI 192.168.203.134@tcp44 [8/256/0/180] [14328.398453] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14328.435575] LNet: Added LNI 192.168.203.134@tcp45 [8/256/0/180] [14338.343554] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14338.413405] LNet: Added LNI 192.168.203.134@tcp46 [8/256/0/180] [14349.506679] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14349.547163] LNet: Added LNI 192.168.203.134@tcp47 [8/256/0/180] [14359.156661] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14359.183565] LNet: Added LNI 192.168.203.134@tcp48 [8/256/0/180] [14368.233305] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14368.269785] LNet: Added LNI 192.168.203.134@tcp49 [8/256/0/180] [14377.778931] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14377.819521] LNet: Added LNI 192.168.203.134@tcp50 [8/256/0/180] [14396.906369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14396.916878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14396.929061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14397.422019] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14397.440576] Lustre: Skipped 1 previous similar message [14397.445975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14397.450951] Lustre: Skipped 2 previous similar messages [14401.816699] Lustre: server umount lustre-MDT0000 complete [14405.901874] LustreError: 472890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453718 with bad export cookie 15179694785020737491 [14405.905147] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14405.928275] LustreError: 472890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14406.216715] Lustre: server umount lustre-MDT0001 complete [14424.032240] Lustre: 472168:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788453720/real 1788453720] req@0000000041385abc x1875329223399424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788453736 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14424.069191] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14426.801723] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [14427.167621] Lustre: 472168:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788453723/real 1788453723] req@0000000039509bda x1875329223399552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788453739 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14429.152348] Lustre: 472170:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788453725/real 1788453725] req@00000000c1e9a26c x1875329223399680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788453741 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14432.223787] Lustre: 472171:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788453728/real 1788453728] req@00000000370dc91a x1875329223399936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788453744 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14436.319218] Lustre: 472169:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788453733/real 1788453733] req@00000000e2a927d3 x1875329223400064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788453749 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14436.361269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14452.109181] LustreError: 474861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14452.137051] LustreError: 474861:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14452.199868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14453.216602] LustreError: 474862:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14453.217855] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14457.507482] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [14458.355714] LustreError: 474861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14463.462555] LustreError: 474862:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14466.436959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14466.808483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14468.859777] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14471.577209] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [14479.226639] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14484.934125] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14502.885023] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14502.899843] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14502.922441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14504.930896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14504.944551] Lustre: Skipped 2 previous similar messages [14508.756665] Lustre: server umount lustre-OST0000 complete [14513.123469] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14513.132504] LustreError: Skipped 1 previous similar message [14513.137599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14513.151601] Lustre: Skipped 1 previous similar message [14513.158030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14519.049243] Lustre: server umount lustre-MDT0000 complete [14520.292202] LustreError: 499846:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14520.342981] LustreError: 499846:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14523.910462] LustreError: 472891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453836 with bad export cookie 15179694785020740039 [14523.915310] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14523.941314] LustreError: 472891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14524.681128] Lustre: server umount lustre-MDT0001 complete [14537.245402] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [14542.445535] Key type lgssc unregistered [14542.922030] LNet: 501894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14542.946900] LNetError: 501894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14542.959275] LNet: Removed LNI 192.168.203.134@tcp [14542.964313] LNet: Removed LNI 192.168.203.134@tcp10 [14542.968210] LNet: Removed LNI 192.168.203.134@tcp11 [14542.971939] LNet: Removed LNI 192.168.203.134@tcp12 [14542.986815] LNet: Removed LNI 192.168.203.134@tcp13 [14542.995314] LNet: Removed LNI 192.168.203.134@tcp14 [14543.003942] LNet: Removed LNI 192.168.203.134@tcp15 [14543.009971] LNet: Removed LNI 192.168.203.134@tcp16 [14543.018879] LNet: Removed LNI 192.168.203.134@tcp17 [14543.023560] LNet: Removed LNI 192.168.203.134@tcp18 [14543.027984] LNet: Removed LNI 192.168.203.134@tcp19 [14543.031768] LNet: Removed LNI 192.168.203.134@tcp20 [14543.034942] LNet: Removed LNI 192.168.203.134@tcp21 [14543.040281] LNet: Removed LNI 192.168.203.134@tcp22 [14543.043602] LNet: Removed LNI 192.168.203.134@tcp23 [14543.051820] LNet: Removed LNI 192.168.203.134@tcp24 [14543.054472] LNet: Removed LNI 192.168.203.134@tcp25 [14543.066263] LNet: Removed LNI 192.168.203.134@tcp26 [14543.078617] LNet: Removed LNI 192.168.203.134@tcp27 [14543.091985] LNet: Removed LNI 192.168.203.134@tcp28 [14543.100975] LNet: Removed LNI 192.168.203.134@tcp29 [14543.120434] LNet: Removed LNI 192.168.203.134@tcp30 [14543.129150] LNet: Removed LNI 192.168.203.134@tcp31 [14543.132257] LNet: Removed LNI 192.168.203.134@tcp32 [14543.136295] LNet: Removed LNI 192.168.203.134@tcp33 [14543.139164] LNet: Removed LNI 192.168.203.134@tcp34 [14543.141943] LNet: Removed LNI 192.168.203.134@tcp35 [14543.145250] LNet: Removed LNI 192.168.203.134@tcp36 [14543.149324] LNet: Removed LNI 192.168.203.134@tcp37 [14543.152500] LNet: Removed LNI 192.168.203.134@tcp38 [14543.155605] LNet: Removed LNI 192.168.203.134@tcp39 [14543.158263] LNet: Removed LNI 192.168.203.134@tcp40 [14543.161180] LNet: Removed LNI 192.168.203.134@tcp41 [14543.163938] LNet: Removed LNI 192.168.203.134@tcp42 [14543.166537] LNet: Removed LNI 192.168.203.134@tcp43 [14543.169023] LNet: Removed LNI 192.168.203.134@tcp44 [14543.171531] LNet: Removed LNI 192.168.203.134@tcp45 [14543.173482] LNet: Removed LNI 192.168.203.134@tcp46 [14543.176379] LNet: Removed LNI 192.168.203.134@tcp47 [14543.178642] LNet: Removed LNI 192.168.203.134@tcp48 [14543.182254] LNet: Removed LNI 192.168.203.134@tcp49 [14543.195596] LNet: Removed LNI 192.168.203.134@tcp50 [14544.130378] Key type .llcrypt unregistered [14544.136361] Key type ._llcrypt unregistered [14566.894335] Key type ._llcrypt registered [14566.897404] Key type .llcrypt registered [14566.997726] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [14578.917421] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [14579.598317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14579.627780] alg: No test for adler32 (adler32-zlib) [14580.621722] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [14580.763807] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [14582.487440] Key type lgssc registered [14583.911237] Lustre: Echo OBD driver; http://www.lustre.org/ [14618.651233] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14618.665080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14619.820797] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14619.839629] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14619.903596] Lustre: lustre-MDT0000: new disk, initializing [14619.992086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14620.005768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14623.658153] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [14632.521838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14632.598759] Lustre: 505067:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14632.657672] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14632.664038] Lustre: Skipped 1 previous similar message [14632.732074] Lustre: lustre-MDT0001: new disk, initializing [14632.833176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14632.867259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14632.875618] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14636.997239] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [14646.534090] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14652.622536] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14661.063968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14661.355682] Lustre: *** cfs_fail_loc=90f, val=0*** [14661.466563] Lustre: lustre-OST0000: new disk, initializing [14661.470321] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14661.495299] Lustre: 507082:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14661.636221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14667.921528] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [14669.420037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14669.428405] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14669.536545] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14677.458452] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14685.160795] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14685.162140] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14685.171992] LustreError: Skipped 1 previous similar message [14685.173041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14685.181195] Lustre: Skipped 1 previous similar message [14685.202557] Lustre: Skipped 1 previous similar message [14690.273816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14690.280563] Lustre: Skipped 1 previous similar message [14690.808422] Lustre: server umount lustre-OST0000 complete [14695.393132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14695.414316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14695.422719] Lustre: Skipped 1 previous similar message [14700.512811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14700.521185] Lustre: Skipped 1 previous similar message [14702.668650] Lustre: server umount lustre-MDT0000 complete [14705.999711] LustreError: 505859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454018 with bad export cookie 1210166109790763926 [14706.009245] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14706.018541] LustreError: 505859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14706.273588] Lustre: server umount lustre-MDT0001 complete [14716.503828] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [14718.497991] Key type lgssc unregistered [14718.743655] LNet: 508984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14718.752342] LNetError: 508984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14718.777835] LNet: Removed LNI 192.168.203.134@tcp [14719.443821] Key type .llcrypt unregistered [14719.449991] Key type ._llcrypt unregistered [14739.265182] Key type ._llcrypt registered [14739.268585] Key type .llcrypt registered [14739.468661] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [14740.385528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14740.546130] alg: No test for adler32 (adler32-zlib) [14741.704547] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [14742.146952] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [14744.071228] Key type lgssc registered [14745.716667] Lustre: Echo OBD driver; http://www.lustre.org/ [14754.225086] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14754.252164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14755.611748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14759.116161] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [14766.050748] LustreError: 510252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14766.073366] LustreError: 510252:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14766.753752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14767.122624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14771.861239] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [14778.907955] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14785.150992] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14792.677174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14792.995463] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14800.522405] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [14807.350828] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14821.346032] loop0: detected capacity change from 0 to 204800000 [14821.565415] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14821.582393] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14821.604637] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14828.158953] loop0: detected capacity change from 0 to 204800000 [14828.291322] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14828.312662] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14828.341945] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14833.180242] loop0: detected capacity change from 0 to 204800000 [14839.084608] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14839.284136] Lustre: fs15246-OST0000: new disk, initializing [14839.288883] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14839.306301] Lustre: 513847:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14839.367466] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14847.283129] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [14859.794571] loop1: detected capacity change from 0 to 204800000 [14866.132071] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14866.236193] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14866.281582] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14866.376526] Lustre: fs15246-MDT0000: new disk, initializing [14866.533985] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14866.556456] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14866.653934] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14867.681118] Lustre: 514970:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000c0b02f30 x1875329998176000/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788454196 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14870.959985] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [14876.791391] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14876.795764] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14876.818727] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14888.432845] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14888.457832] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14893.549932] LustreError: 510248:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14893.640292] Lustre: server umount fs15246-MDT0000 complete [14898.668138] LustreError: 510248:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14901.891404] Lustre: server umount fs15246-OST0000 complete [14914.016548] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14914.024736] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14914.027716] LustreError: Skipped 1 previous similar message [14914.028629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14914.057736] Lustre: Skipped 1 previous similar message [14918.420497] Lustre: server umount lustre-OST0000 complete [14924.259129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14924.278691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14924.286515] Lustre: Skipped 2 previous similar messages [14928.331400] Lustre: server umount lustre-MDT0000 complete [14929.379623] LustreError: 511257:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14929.403768] LustreError: 511257:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14932.327034] LustreError: 510231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454245 with bad export cookie 18361031945525942505 [14932.332412] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14932.337305] LustreError: 510231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14932.693719] Lustre: server umount lustre-MDT0001 complete [14943.815768] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [14947.900617] Key type lgssc unregistered [14948.369886] LNet: 518063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14948.388756] LNetError: 518063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14948.427976] LNet: Removed LNI 192.168.203.134@tcp [14949.450344] Key type .llcrypt unregistered [14949.452809] Key type ._llcrypt unregistered [14968.192686] Key type ._llcrypt registered [14968.194965] Key type .llcrypt registered [14968.310518] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [14969.482167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14969.510224] alg: No test for adler32 (adler32-zlib) [14970.513973] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [14970.702093] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [14972.319181] Key type lgssc registered [14973.564554] Lustre: Echo OBD driver; http://www.lustre.org/ [15013.585966] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [15023.672881] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [15055.247779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15055.282395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15056.453088] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15056.475651] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15056.520855] Lustre: lustre-MDT0000: new disk, initializing [15056.583085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15056.616257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15061.937658] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [15070.166752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15070.318579] Lustre: 522524:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15070.396495] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15070.407681] Lustre: Skipped 1 previous similar message [15070.500631] Lustre: lustre-MDT0001: new disk, initializing [15070.634507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15070.665144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15070.682149] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15075.365976] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [15084.585563] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15089.848854] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15098.161968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15098.536335] Lustre: lustre-OST0000: new disk, initializing [15098.540403] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15098.548452] Lustre: 524497:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15098.638389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15105.981250] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [15108.693381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15108.705970] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15108.800411] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15115.676256] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15120.229342] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15120.457443] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15124.306418] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15124.576046] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15129.580766] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15129.584432] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15129.591161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15129.600299] LustreError: Skipped 1 previous similar message [15129.640052] Lustre: Skipped 1 previous similar message [15132.676252] Lustre: server umount lustre-OST0000 complete [15137.250719] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15137.258756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15137.276952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15137.281069] Lustre: Skipped 1 previous similar message [15139.810870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15139.812227] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15143.096469] Lustre: server umount lustre-MDT0000 complete [15147.152541] LustreError: 522516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454459 with bad export cookie 8273568656421196567 [15147.162549] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15147.169588] LustreError: 522516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15147.514822] Lustre: server umount lustre-MDT0001 complete [15173.674022] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [15184.022202] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [15215.962508] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14916 sec ========= 12:55:25 (1788454525) [15217.679634] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:55:27 (1788454527) === [15221.370169] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:55:31 (1788454531) === [15261.379808] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [15264.366475] Key type lgssc unregistered [15264.657267] LNet: 531141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15264.670157] LNetError: 531141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15264.688425] LNet: Removed LNI 192.168.203.134@tcp [15265.730198] Key type .llcrypt unregistered [15265.733353] Key type ._llcrypt unregistered