[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 518572305 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002384] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009143] LSM: Security Framework initializing [ 0.010051] Yama: becoming mindful. [ 0.011044] SELinux: Initializing. [ 0.012092] *** VALIDATE selinux *** [ 0.021253] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026297] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027134] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028097] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029125] *** VALIDATE tmpfs *** [ 0.031149] *** VALIDATE proc *** [ 0.032169] *** VALIDATE cgroup *** [ 0.033000] *** VALIDATE cgroup2 *** [ 0.033327] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034179] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037027] Spectre V2 : User space: Vulnerable [ 0.038009] Speculative Store Bypass: Vulnerable [ 0.041043] debug: unmapping init [mem 0xffffffff90059000-0xffffffff90060fff] [ 0.043205] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044736] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045025] ... version: 2 [ 0.046015] ... bit width: 48 [ 0.047010] ... generic registers: 4 [ 0.048008] ... value mask: 0000ffffffffffff [ 0.049018] ... max period: 00007fffffffffff [ 0.050016] ... fixed-purpose events: 3 [ 0.050955] ... event mask: 000000070000000f [ 0.051306] rcu: Hierarchical SRCU implementation. [ 0.053528] smp: Bringing up secondary CPUs ... [ 0.054649] x86: Booting SMP configuration: [ 0.055026] .... node #0, CPUs: #1 #2 #3 [ 0.058412] smp: Brought up 1 node, 4 CPUs [ 0.060015] smpboot: Max logical packages: 1 [ 0.061021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.181295] node 0 deferred pages initialised in 119ms [ 0.185009] devtmpfs: initialized [ 0.186343] x86/mm: Memory block size: 128MB [ 0.189023] gcov: version magic: 0x41383552 [ 0.191366] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.195024] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.196268] pinctrl core: initialized pinctrl subsystem [ 0.197184] [ 0.197681] ************************************************************* [ 0.198015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.199012] ** ** [ 0.200023] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.201018] ** ** [ 0.202016] ** This means that this kernel is built to expose internal ** [ 0.203020] ** IOMMU data structures, which may compromise security on ** [ 0.204014] ** your system. ** [ 0.205016] ** ** [ 0.206016] ** If you see this message and you are not debugging the ** [ 0.207016] ** kernel, report this immediately to your vendor! ** [ 0.208014] ** ** [ 0.209015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.210011] ************************************************************* [ 0.211793] NET: Registered protocol family 16 [ 0.212499] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.213075] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.214080] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.216039] cpuidle: using governor menu [ 0.217000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.220091] PCI: Using configuration type 1 for base access [ 0.221121] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.231053] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.233167] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.236186] cryptd: max_cpu_qlen set to 1000 [ 0.239288] ACPI: Added _OSI(Module Device) [ 0.240020] ACPI: Added _OSI(Processor Device) [ 0.241015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.243016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.248202] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.255524] ACPI: Interpreter enabled [ 0.257071] ACPI: PM: (supports S0 S3 S4 S5) [ 0.258019] ACPI: Using IOAPIC for interrupt routing [ 0.259136] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.263403] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.272594] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.274064] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.277030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.281125] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.286723] acpiphp: Slot [2] registered [ 0.288267] acpiphp: Slot [5] registered [ 0.290196] acpiphp: Slot [6] registered [ 0.292186] acpiphp: Slot [7] registered [ 0.293235] acpiphp: Slot [8] registered [ 0.295188] acpiphp: Slot [9] registered [ 0.297188] acpiphp: Slot [10] registered [ 0.298232] acpiphp: Slot [3] registered [ 0.300143] acpiphp: Slot [4] registered [ 0.301138] acpiphp: Slot [11] registered [ 0.303140] acpiphp: Slot [12] registered [ 0.304227] acpiphp: Slot [13] registered [ 0.306152] acpiphp: Slot [14] registered [ 0.308173] acpiphp: Slot [15] registered [ 0.309136] acpiphp: Slot [16] registered [ 0.311140] acpiphp: Slot [17] registered [ 0.312224] acpiphp: Slot [18] registered [ 0.314123] acpiphp: Slot [19] registered [ 0.315125] acpiphp: Slot [20] registered [ 0.316059] acpiphp: Slot [21] registered [ 0.317116] acpiphp: Slot [22] registered [ 0.318000] acpiphp: Slot [23] registered [ 0.320079] acpiphp: Slot [24] registered [ 0.321080] acpiphp: Slot [25] registered [ 0.322092] acpiphp: Slot [26] registered [ 0.324110] acpiphp: Slot [27] registered [ 0.325117] acpiphp: Slot [28] registered [ 0.327126] acpiphp: Slot [29] registered [ 0.328110] acpiphp: Slot [30] registered [ 0.329092] acpiphp: Slot [31] registered [ 0.331066] PCI host bridge to bus 0000:00 [ 0.332037] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.334037] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.337042] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.340049] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.342027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.343024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.345224] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.349000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.351049] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.359957] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.364992] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.367021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.369018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.372027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.375075] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.377848] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.380057] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.382975] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.389022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.403071] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.409932] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.415736] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.426019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.434023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.461020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.468649] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.479025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.488025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.514023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.525288] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.531020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.538019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.551022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.558065] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.567026] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.574018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.584020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.594613] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.602025] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.609023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.625025] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.636942] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.642021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.648026] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.660016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.670880] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.673547] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.677195] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.680457] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.683398] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.688142] iommu: Default domain type: Passthrough [ 0.690701] SCSI subsystem initialized [ 0.691159] ACPI: bus type USB registered [ 0.693098] usbcore: registered new interface driver usbfs [ 0.695097] usbcore: registered new interface driver hub [ 0.697081] usbcore: registered new device driver usb [ 0.698194] pps_core: LinuxPPS API ver. 1 registered [ 0.700010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.703063] PTP clock support registered [ 0.705086] EDAC MC: Ver: 3.0.0 [ 0.706244] PCI: Using ACPI for IRQ routing [ 0.707960] NetLabel: Initializing [ 0.709012] NetLabel: domain hash size = 128 [ 0.710007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.712082] NetLabel: unlabeled traffic allowed by default [ 0.713286] vgaarb: loaded [ 0.715177] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.716010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.722386] clocksource: Switched to clocksource kvm-clock [ 0.833672] VFS: Disk quotas dquot_6.6.0 [ 0.834958] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.836858] *** VALIDATE ramfs *** [ 0.837830] *** VALIDATE hugetlbfs *** [ 0.839927] pnp: PnP ACPI init [ 0.842209] pnp: PnP ACPI: found 6 devices [ 0.858163] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.860424] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.864671] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.866935] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.869614] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.872219] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.875398] NET: Registered protocol family 2 [ 0.878228] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.883934] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.887742] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.892842] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.897325] TCP: Hash tables configured (established 65536 bind 65536) [ 0.900468] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.903886] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.907113] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.910354] NET: Registered protocol family 1 [ 0.913058] RPC: Registered named UNIX socket transport module. [ 0.915241] RPC: Registered udp transport module. [ 0.917238] RPC: Registered tcp transport module. [ 0.919096] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.921188] NET: Registered protocol family 44 [ 0.922793] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.924539] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.926514] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.928519] PCI: CLS 0 bytes, default 64 [ 0.930112] Unpacking initramfs... [ 2.341427] debug: unmapping init [mem 0xffffa0edfcc54000-0xffffa0edfffbffff] [ 2.345144] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.346941] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.349616] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.846947] Initialise system trusted keyrings [ 2.848615] Key type blacklist registered [ 2.850397] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.859648] zbud: loaded [ 2.862884] *** VALIDATE nfs *** [ 2.864239] *** VALIDATE nfs4 *** [ 2.865741] pstore: using deflate compression [ 2.868378] Platform Keyring initialized [ 2.983324] NET: Registered protocol family 38 [ 2.985486] Key type asymmetric registered [ 2.987391] Asymmetric key parser 'x509' registered [ 2.989666] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.993113] io scheduler mq-deadline registered [ 2.994927] io scheduler kyber registered [ 2.996824] io scheduler bfq registered [ 2.998822] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.001798] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.005053] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.007845] ACPI: Power Button [PWRF] [ 3.014203] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.023762] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.036525] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.043136] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.056561] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.084167] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.112103] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.116727] Non-volatile memory driver v1.3 [ 3.118689] Linux agpgart interface v0.103 [ 3.149137] virtio_blk virtio1: [vda] 149376 512-byte logical blocks (76.5 MB/72.9 MiB) [ 3.151480] vda: detected capacity change from 0 to 76480512 [ 3.164538] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.167152] vdb: detected capacity change from 0 to 1073741824 [ 3.179981] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.183863] vdc: detected capacity change from 0 to 2621440000 [ 3.203464] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.206646] vdd: detected capacity change from 0 to 2621440000 [ 3.220148] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.223215] vde: detected capacity change from 0 to 4294967296 [ 3.236983] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.240219] vdf: detected capacity change from 0 to 4294967296 [ 3.247107] libphy: Fixed MDIO Bus: probed [ 3.259815] usbcore: registered new interface driver usbserial_generic [ 3.261870] usbserial: USB Serial support registered for generic [ 3.264121] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.267655] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.269050] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.271514] mousedev: PS/2 mouse device common for all mice [ 3.273653] rtc_cmos 00:05: RTC can wake from S4 [ 3.275477] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.277322] rtc_cmos 00:05: registered as rtc0 [ 3.281569] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.282062] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.286450] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.288383] intel_pstate: CPU model not supported [ 3.294940] hid: raw HID events driver (C) Jiri Kosina [ 3.297200] usbcore: registered new interface driver usbhid [ 3.299422] usbhid: USB HID core driver [ 3.301088] drop_monitor: Initializing network drop monitor service [ 3.303779] Initializing XFRM netlink socket [ 3.305987] NET: Registered protocol family 10 [ 3.309154] Segment Routing with IPv6 [ 3.310448] NET: Registered protocol family 17 [ 3.312311] mpls_gso: MPLS GSO support [ 3.318557] RAS: Correctable Errors collector initialized. [ 3.320380] AVX version of gcm_enc/dec engaged. [ 3.321621] AES CTR mode by8 optimization enabled [ 3.397761] sched_clock: Marking stable (3397744338, 0)->(4350238484, -952494146) [ 3.400188] registered taskstats version 1 [ 3.401834] Loading compiled-in X.509 certificates [ 3.404170] zswap: loaded using pool lzo/zbud [ 3.433645] Key type big_key registered [ 3.448244] Key type encrypted registered [ 3.449822] ima: No TPM chip found, activating TPM-bypass! [ 3.451368] ima: Allocated hash algorithm: sha1 [ 3.452582] ima: No architecture policies found [ 3.453899] evm: Initialising EVM extended attributes: [ 3.455404] evm: security.selinux [ 3.456321] evm: security.ima [ 3.457161] evm: security.capability [ 3.458196] evm: HMAC attrs: 0x1 [ 3.460739] rtc_cmos 00:05: setting system clock to 2026-08-10 22:05:36 UTC (1786399536) [ 3.466653] debug: unmapping init [mem 0xffffffff91003000-0xffffffff911fffff] [ 3.469171] debug: unmapping init [mem 0xffffffff8fd82000-0xffffffff90058fff] [ 3.477064] Write protecting the kernel read-only data: 28672k [ 3.480479] debug: unmapping init [mem 0xffffffff8e403000-0xffffffff8e5fffff] [ 3.482574] debug: unmapping init [mem 0xffffffff8ed14000-0xffffffff8edfffff] [ 3.515576] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.521718] systemd[1]: Detected virtualization kvm. [ 3.523488] systemd[1]: Detected architecture x86-64. [ 3.524836] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.553090] systemd[1]: No hostname configured. [ 3.554669] systemd[1]: Set hostname to . [ 3.556328] random: systemd: uninitialized urandom read (16 bytes read) [ 3.558090] systemd[1]: Initializing machine ID from random generator. [ 3.686720] random: systemd: uninitialized urandom read (16 bytes read) [ 3.689103] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.693128] random: systemd: uninitialized urandom read (16 bytes read) [ 3.695717] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.702208] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.271291] device-mapper: uevent: version 1.0.3 [ 4.273427] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.000982] virtio_net virtio0 ens2: renamed from eth0 [ 5.024308] random: fast init done [ 5.122694] scsi host0: ata_piix [ 5.148117] scsi host1: ata_piix [ 5.150291] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.152053] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.371645] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.963792] random: crng init done [ 9.965169] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.310190] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.498698] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.732315] SELinux: Disabled at runtime. [ 11.795915] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.803705] systemd[1]: Detected virtualization kvm. [ 11.805563] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.331415] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.335721] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.340430] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.345547] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.349564] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.357527] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.364912] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ 12.549838] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.865170] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.220789] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.227330] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.454267] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.470243] EDAC sbridge: Ver: 1.1.2 [ 15.175908] Key type dns_resolver registered [ 15.488129] NFS: Registering the id_resolver key type [ 15.490234] Key type id_resolver registered [ 15.491854] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg204-server login: [ 40.909869] libcfs: loading out-of-tree module taints kernel. [ 40.930688] Key type ._llcrypt registered [ 40.932234] Key type .llcrypt registered [ 40.986679] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 49.394734] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 50.085654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 50.094079] alg: No test for adler32 (adler32-zlib) [ 51.120041] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 51.458499] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 53.095154] Key type lgssc registered [ 54.093543] Lustre: Echo OBD driver; http://www.lustre.org/ [ 73.286402] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 87.232559] hrtimer: interrupt took 4719894 ns [ 113.228885] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 127.552247] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 127.599871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 128.925717] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 128.977736] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 129.109248] Lustre: lustre-MDT0000: new disk, initializing [ 129.224793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 129.244782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 134.744451] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 147.871620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 147.991421] Lustre: 6483:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 148.024453] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 148.028996] Lustre: Skipped 1 previous similar message [ 148.121243] Lustre: lustre-MDT0001: new disk, initializing [ 148.208354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 148.235487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 148.245515] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 153.018697] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 158.172512] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 168.477518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 168.813522] Lustre: lustre-OST0000: new disk, initializing [ 168.817324] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 168.823621] Lustre: 8418:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 168.902590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 173.031585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 173.051584] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 173.115689] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 175.825497] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 191.939552] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 192.115400] Lustre: lustre-OST0001: new disk, initializing [ 192.118510] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 192.125597] Lustre: 9490:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 192.204859] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 199.252623] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 201.372172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 201.379969] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 201.437913] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 211.898727] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 221.919408] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 228.471647] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing check_logdir /tmp/testlogs/ [ 233.880994] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing yml_node [ 238.100661] Lustre: DEBUG MARKER: Client: 2.17.54.107 [ 240.983653] Lustre: DEBUG MARKER: MDS: 2.17.54.107 [ 243.617470] Lustre: DEBUG MARKER: OSS: 2.17.54.107 [ 245.140564] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 10 18:09:36 EDT 2026 [ 262.464460] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 264.054261] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 283.620976] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 283.624169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 283.646067] Lustre: Skipped 1 previous similar message [ 283.670925] Lustre: Skipped 3 previous similar messages [ 287.974973] Lustre: server umount lustre-MDT0000 complete [ 292.135911] LustreError: 6474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786399825 with bad export cookie 17349485803496796946 [ 292.152387] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 292.178377] LustreError: 6474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 293.856796] LustreError: 10068:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 293.858772] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 293.884610] LustreError: 10068:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 293.906243] Lustre: Skipped 2 previous similar messages [ 293.909991] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 293.923647] Lustre: Skipped 1 previous similar message [ 297.889051] LustreError: 10068:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 297.902435] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 297.921831] LustreError: 10068:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 297.931568] Lustre: Skipped 1 previous similar message [ 298.488604] Lustre: server umount lustre-MDT0001 complete [ 302.455808] Lustre: server umount lustre-OST0000 complete [ 306.293720] Lustre: server umount lustre-OST0001 complete [ 312.320618] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 323.393395] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 362.254108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 362.590874] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 362.624431] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 362.737573] Lustre: lustre-MDT0000: new disk, initializing [ 362.832224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 362.843450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 366.694752] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 375.719794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 375.840582] Lustre: 15045:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 375.949048] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 375.954483] Lustre: Skipped 1 previous similar message [ 376.067286] Lustre: lustre-MDT0001: new disk, initializing [ 376.154239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 376.205801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 376.223674] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 380.031131] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 387.893900] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 392.857734] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 399.032039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 399.389234] Lustre: lustre-OST0000: new disk, initializing [ 399.393232] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 399.400513] Lustre: 17011:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 399.497778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 400.905833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 400.916934] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 401.048785] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 405.717793] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 414.958873] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 420.336787] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 420.669723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 425.708190] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 425.920444] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 431.587359] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 431.589384] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 431.595875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 431.604581] Lustre: Skipped 2 previous similar messages [ 436.705201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 436.714044] Lustre: Skipped 2 previous similar messages [ 442.336068] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 442.494183] Lustre: server umount lustre-OST0000 complete [ 446.944917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 446.955869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 446.962074] Lustre: Skipped 2 previous similar messages [ 452.764877] LustreError: 18553:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffffa0ee459d4e00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 453.100975] Lustre: server umount lustre-MDT0000 complete [ 457.185798] LustreError: 18352:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 457.201904] LustreError: 18352:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 457.212380] LustreError: 15037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786399990 with bad export cookie 17349485803496798479 [ 457.219588] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 457.252340] LustreError: 15037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 457.995628] Lustre: server umount lustre-MDT0001 complete [ 463.075868] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 18:13:14 (1786399994) [ 465.867791] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 467.630090] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 18:13:18 (1786399998) [ 474.274962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 475.019680] LustreError: 19425:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 475.116159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 479.775541] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 480.227484] LustreError: 19426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 489.461716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 495.002431] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 501.661953] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 508.124757] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 516.154121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 524.982755] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 531.734410] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 544.739298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 544.740573] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 544.750292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 544.750301] Lustre: Skipped 2 previous similar messages [ 544.767241] Lustre: Skipped 2 previous similar messages [ 549.541851] Lustre: server umount lustre-OST0000 complete [ 554.978154] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 568.287327] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 568.628771] Lustre: server umount lustre-MDT0000 complete [ 569.314496] LustreError: 19420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 569.320926] LustreError: 19420:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 572.610434] LustreError: 19406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786400105 with bad export cookie 17349485803496799354 [ 572.614829] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 572.618417] LustreError: 19406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 573.040820] Lustre: server umount lustre-MDT0001 complete [ 586.022083] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 589.587151] Key type lgssc unregistered [ 589.957739] LNet: 22958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 589.965042] LNetError: 22958:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 591.033132] LNet: Removed LNI 192.168.202.104@tcp [ 592.039162] Key type .llcrypt unregistered [ 592.043300] Key type ._llcrypt unregistered [ 615.492579] Key type ._llcrypt registered [ 615.495695] Key type .llcrypt registered [ 615.614274] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 617.626736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 617.744486] alg: No test for adler32 (adler32-zlib) [ 619.164211] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 619.577309] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 621.415198] Key type lgssc registered [ 622.771809] Lustre: Echo OBD driver; http://www.lustre.org/ [ 632.943520] loop: module loaded [ 633.066316] loop0: detected capacity change from 0 to 204800000 [ 633.248302] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 633.253665] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 633.535550] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 637.977510] loop0: detected capacity change from 0 to 204800000 [ 638.175283] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 638.184102] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 638.191379] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 642.518947] loop0: detected capacity change from 0 to 204800000 [ 647.765098] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 647.810544] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 649.161649] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 649.276718] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 649.437800] Lustre: lustre-MDT0000: new disk, initializing [ 649.602218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 649.659983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 656.025126] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 665.844319] loop1: detected capacity change from 0 to 204800000 [ 671.936250] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 672.242502] Lustre: lustre-OST07c6: new disk, initializing [ 672.247291] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 672.251442] Lustre: Skipped 1 previous similar message [ 672.255420] Lustre: 26151:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 672.335220] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 679.028053] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 679.526591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 679.542535] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 679.591670] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 684.890725] Lustre: 25136:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 684.899661] Lustre: 25136:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 684.909609] Lustre: 25136:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 684.917227] Lustre: 25136:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 684.924212] Lustre: 25136:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 684.931142] Lustre: 25136:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 698.619664] Lustre: 25137:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 698.626141] Lustre: 25137:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 698.632039] Lustre: 25137:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 698.636340] Lustre: 25137:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 698.641961] Lustre: 25137:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 698.645426] Lustre: 25137:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 703.388951] Lustre: 25137:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 703.400564] Lustre: 25137:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 703.413481] Lustre: 25137:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 703.427658] Lustre: 25137:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 703.439821] Lustre: 25137:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 703.453457] Lustre: 25137:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 703.470166] Lustre: 25137:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 703.496498] Lustre: 25137:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 703.507127] Lustre: 25137:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 703.516050] Lustre: 25137:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 703.527087] Lustre: 25137:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 703.533037] Lustre: 25137:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 706.778751] Lustre: 25136:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 706.789301] Lustre: 25136:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 706.801413] Lustre: 25136:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 706.810141] Lustre: 25136:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 706.822284] Lustre: 25136:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 706.833192] Lustre: 25136:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 706.840430] Lustre: 25136:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 706.846276] Lustre: 25136:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 706.850736] Lustre: 25136:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 706.855646] Lustre: 25136:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 706.860245] Lustre: 25136:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 706.864327] Lustre: 25136:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 712.063964] Lustre: 26159:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 712.072130] Lustre: 26159:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 712.083551] Lustre: 26159:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 712.091095] Lustre: 26159:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 712.101281] Lustre: 26159:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 712.114064] Lustre: 26159:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 712.131143] Lustre: 26159:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 712.148670] Lustre: 26159:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 712.163141] Lustre: 26159:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 712.173269] Lustre: 26159:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 712.181396] Lustre: 26159:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 712.192211] Lustre: 26159:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 720.435764] Lustre: 26159:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 720.443123] Lustre: 26159:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 720.448167] Lustre: 26159:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 720.454207] Lustre: 26159:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 720.459921] Lustre: 26159:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 720.467077] Lustre: 26159:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 720.473278] Lustre: 26159:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 720.481643] Lustre: 26159:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 720.485801] Lustre: 26159:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 720.492909] Lustre: 26159:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 720.498930] Lustre: 26159:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 720.504067] Lustre: 26159:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 776.673959] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 776.693076] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 778.450326] Lustre: server umount lustre-OST07c6 complete [ 788.371369] Lustre: server umount lustre-MDT0000 complete [ 806.178382] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 809.397984] Key type lgssc unregistered [ 809.657165] LNet: 28707:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 809.662234] LNetError: 28707:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 809.678250] LNet: Removed LNI 192.168.202.104@tcp [ 810.777828] Key type .llcrypt unregistered [ 810.786551] Key type ._llcrypt unregistered [ 829.641604] Key type ._llcrypt registered [ 829.643590] Key type .llcrypt registered [ 829.762901] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 830.960258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 831.128636] alg: No test for adler32 (adler32-zlib) [ 832.219675] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 832.503988] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 834.223679] Key type lgssc registered [ 835.416397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 843.619673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 843.644374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 845.140669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 850.154718] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 855.527556] LustreError: 29978:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 855.562145] LustreError: 29978:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 860.133342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 860.456734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 864.762338] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 873.530191] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 881.104354] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 888.992206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 889.339850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 890.367061] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 896.890760] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 904.340129] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 921.060963] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 921.071336] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 921.095032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 923.167413] Lustre: server umount lustre-OST0000 complete [ 931.299586] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 931.303443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 931.319753] Lustre: Skipped 2 previous similar messages [ 931.344807] Lustre: Skipped 2 previous similar messages [ 933.843505] Lustre: server umount lustre-MDT0000 complete [ 936.418926] LustreError: 31448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 936.439911] LustreError: 31448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 937.885789] LustreError: 29958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786400470 with bad export cookie 10251495387692131464 [ 937.892422] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 937.904118] LustreError: 29958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 938.233174] Lustre: server umount lustre-MDT0001 complete [ 950.849804] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 953.842706] Key type lgssc unregistered [ 954.125520] LNet: 33719:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 954.133756] LNetError: 33719:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 955.175491] LNet: Removed LNI 192.168.202.104@tcp [ 956.176462] Key type .llcrypt unregistered [ 956.178979] Key type ._llcrypt unregistered [ 976.660541] Key type ._llcrypt registered [ 976.663989] Key type .llcrypt registered [ 976.833931] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 978.097307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 978.117254] alg: No test for adler32 (adler32-zlib) [ 979.213322] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 979.414290] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 981.096334] Key type lgssc registered [ 982.264637] Lustre: Echo OBD driver; http://www.lustre.org/ [ 991.649619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 991.691481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 993.229146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 998.970657] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1003.490060] LustreError: 34991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1003.544931] LustreError: 34991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1008.620257] LustreError: 34992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1010.800255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1011.283548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1017.127451] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1026.689254] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1033.397807] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1041.773058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1042.253710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1050.233634] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1058.859824] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1063.814312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1077.232123] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1077.242098] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1077.254620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1079.264231] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1079.271195] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1079.344917] Lustre: server umount lustre-OST0000 complete [ 1084.388186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1084.390060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1084.400369] Lustre: Skipped 1 previous similar message [ 1084.419029] Lustre: Skipped 1 previous similar message [ 1089.514363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1089.531124] Lustre: Skipped 1 previous similar message [ 1090.333438] Lustre: server umount lustre-MDT0000 complete [ 1094.637784] LustreError: 34992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1094.672519] LustreError: 34992:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1096.185175] LustreError: 34972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786400629 with bad export cookie 14327376557791773852 [ 1096.199817] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1096.210083] LustreError: 34972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1096.751556] Lustre: server umount lustre-MDT0001 complete [ 1110.704909] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 1114.966506] Key type lgssc unregistered [ 1115.333884] LNet: 38549:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1115.344601] LNetError: 38549:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1115.361571] LNet: Removed LNI 192.168.202.104@tcp [ 1116.050121] Key type .llcrypt unregistered [ 1116.051715] Key type ._llcrypt unregistered [ 1144.875725] Key type ._llcrypt registered [ 1144.878963] Key type .llcrypt registered [ 1145.040210] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 1146.930858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1146.994373] alg: No test for adler32 (adler32-zlib) [ 1148.166937] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 1148.452933] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 1150.151229] Key type lgssc registered [ 1151.421175] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1160.066826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1160.089221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1161.681162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1168.007933] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1171.940406] LustreError: 39969:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1171.957039] LustreError: 39969:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1177.066533] LustreError: 39970:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1178.241319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1178.656435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1183.757175] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1192.320372] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1198.402418] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1206.931912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1207.459308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1215.082787] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1223.121841] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1231.040140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1240.033876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1240.043844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1240.061121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1241.578934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1241.590833] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1241.614492] Lustre: Skipped 1 previous similar message [ 1244.575632] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.4@tcp (stopping) [ 1244.590802] Lustre: Skipped 2 previous similar messages [ 1246.692854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1251.811885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1251.814975] Lustre: Skipped 3 previous similar messages [ 1254.367640] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1254.695892] Lustre: server umount lustre-MDT0000 complete [ 1254.778234] LustreError: 42274:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1256.937118] LustreError: 39965:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1256.978982] LustreError: 39965:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1258.791860] LustreError: 40703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786400791 with bad export cookie 1169565105003895755 [ 1258.794669] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1258.807443] LustreError: 40703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1259.940135] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.4@tcp (stopping) [ 1262.049121] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1262.054969] LustreError: 39964:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1262.099351] LustreError: 39964:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1270.132621] LustreError: 39964:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1270.155859] LustreError: 39964:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1273.312756] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1273.808966] Lustre: server umount lustre-MDT0001 complete [ 1289.562236] Lustre: server umount lustre-OST0000 complete [ 1310.825496] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 1314.413278] Key type lgssc unregistered [ 1314.955724] LNet: 43855:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1314.964428] LNetError: 43855:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1314.979849] LNet: Removed LNI 192.168.202.104@tcp [ 1316.332767] Key type .llcrypt unregistered [ 1316.335732] Key type ._llcrypt unregistered [ 1345.740779] Key type ._llcrypt registered [ 1345.742146] Key type .llcrypt registered [ 1345.901750] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 1347.494893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1347.508860] alg: No test for adler32 (adler32-zlib) [ 1348.709365] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 1349.028083] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 1350.751936] Key type lgssc registered [ 1352.585507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1362.031863] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1362.071352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1363.840635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1369.983725] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1373.157049] LustreError: 45274:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1373.187721] LustreError: 45274:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1378.279291] LustreError: 45275:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1381.591846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1382.050227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1387.379586] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1397.967473] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1406.201660] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1414.852231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1415.189253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1423.516991] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1433.435712] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1433.740046] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1441.693502] Lustre: lustre-OST0000: Not available for connect from 192.168.202.4@tcp (stopping) [ 1443.302943] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1443.319241] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1443.353028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1444.328587] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1444.333960] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1444.356416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1444.358849] Lustre: Skipped 1 previous similar message [ 1446.776433] LustreError: 47100:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1446.977076] Lustre: server umount lustre-OST0000 complete [ 1468.900122] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1468.908921] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1468.919447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1473.334611] Lustre: server umount lustre-MDT0000 complete [ 1475.041671] LustreError: 46261:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1475.069767] LustreError: 46261:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1478.084677] LustreError: 45255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786401011 with bad export cookie 17880182635435650152 [ 1478.099531] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1478.107704] LustreError: 45255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1484.776476] Lustre: server umount lustre-MDT0001 complete [ 1498.301287] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 1501.660944] Key type lgssc unregistered [ 1502.062632] LNet: 48986:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1502.073550] LNetError: 48986:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1502.103646] LNet: Removed LNI 192.168.202.104@tcp [ 1503.118628] Key type .llcrypt unregistered [ 1503.120981] Key type ._llcrypt unregistered [ 1533.412919] Key type ._llcrypt registered [ 1533.420443] Key type .llcrypt registered [ 1533.566974] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 1535.385782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1535.432620] alg: No test for adler32 (adler32-zlib) [ 1536.707803] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 1537.091026] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 1538.871269] Key type lgssc registered [ 1540.312532] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1550.512196] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1550.549131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1552.226301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1556.992986] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1562.612917] LustreError: 50405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1562.649499] LustreError: 50405:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1567.714365] LustreError: 50404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1567.756634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1568.156890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1573.336912] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1582.346659] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1587.981527] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1597.767802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1597.872395] LustreError: 52225:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1598.107579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1603.576658] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1606.802965] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1615.890556] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1622.164582] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1626.678909] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1639.593818] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-1018h32m49s [ 1641.650909] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1644.516741] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1644.530458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1644.541016] Lustre: Skipped 2 previous similar messages [ 1644.573210] Lustre: Skipped 2 previous similar messages [ 1645.666509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1648.524400] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.4@tcp (stopping) [ 1648.536645] Lustre: Skipped 2 previous similar messages [ 1652.205801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1652.215652] Lustre: Skipped 3 previous similar messages [ 1658.335151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1658.348618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1658.363780] Lustre: Skipped 1 previous similar message [ 1658.843657] Lustre: server umount lustre-MDT0000 complete [ 1661.817240] LustreError: 50399:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1662.773382] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1667.968809] LustreError: 50399:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1667.999662] LustreError: 50399:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1670.093672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1670.196467] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1670.383538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1670.433477] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1671.407230] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1674.799094] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1678.563338] Lustre: DEBUG MARKER: Wait for df (36588) ... [ 1679.902131] Lustre: DEBUG MARKER: done [ 1685.481316] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1685.488965] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1685.523229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1685.526268] Lustre: Skipped 2 previous similar messages [ 1685.994959] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1698.271187] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1698.890518] Lustre: server umount lustre-OST0000 complete [ 1707.489254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1707.490652] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1707.491604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1707.491609] Lustre: Skipped 7 previous similar messages [ 1707.507360] Lustre: Skipped 2 previous similar messages [ 1711.330564] Lustre: server umount lustre-MDT0000 complete [ 1714.657959] LustreError: 51132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1714.671978] LustreError: 51132:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1715.183158] LustreError: 50386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786401248 with bad export cookie 14211517704494718732 [ 1715.186355] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1715.198257] LustreError: 50386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1715.588209] Lustre: server umount lustre-MDT0001 complete [ 1729.703328] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 1733.131554] Key type lgssc unregistered [ 1733.399071] LNet: 55332:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1733.406686] LNetError: 55332:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1733.421283] LNet: Removed LNI 192.168.202.104@tcp [ 1734.453468] Key type .llcrypt unregistered [ 1734.459594] Key type ._llcrypt unregistered [ 1781.557253] Key type ._llcrypt registered [ 1781.559026] Key type .llcrypt registered [ 1781.721898] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 1783.411808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1783.452638] alg: No test for adler32 (adler32-zlib) [ 1784.815854] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 1785.118758] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 1786.799400] Key type lgssc registered [ 1788.194791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1796.253734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1796.269653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1797.491042] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1797.503415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1797.615147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1802.660783] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1812.063395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1812.181974] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1812.215814] Lustre: 57545:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1812.526218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1817.970279] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1827.246655] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1834.133037] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1841.761282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1842.019408] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1842.267252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1848.954830] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1850.056971] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1857.503582] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1866.371745] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-1018h36m37s [ 1868.812678] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1877.518389] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1880.515417] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1880.575107] LustreError: 60138:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1900.978975] Lustre: lustre-MDT0000: Client eb83c1ab-c823-4786-b9da-302608047e09 (at 192.168.202.4@tcp) reconnecting [ 1920.584354] LustreError: 60138:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1922.697757] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1924.831815] Lustre: DEBUG MARKER: done [ 1931.744307] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1931.759657] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1931.791744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1932.262543] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1932.266781] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1932.785171] Lustre: server umount lustre-OST0000 complete [ 1937.378637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1937.398962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1937.403340] Lustre: Skipped 3 previous similar messages [ 1938.480102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1938.483886] Lustre: Skipped 1 previous similar message [ 1942.651845] Lustre: server umount lustre-MDT0000 complete [ 1943.524987] LustreError: 60628:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1943.544423] LustreError: 60628:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1946.731686] LustreError: 57539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786401479 with bad export cookie 9092152319519615273 [ 1946.736501] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1946.750718] LustreError: 57539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1947.145928] Lustre: server umount lustre-MDT0001 complete [ 1957.888634] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 1960.293118] Key type lgssc unregistered [ 1960.629613] LNet: 61878:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1960.634288] LNetError: 61878:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1960.674103] LNet: Removed LNI 192.168.202.104@tcp [ 1961.292303] Key type .llcrypt unregistered [ 1961.296795] Key type ._llcrypt unregistered [ 2008.423585] Key type ._llcrypt registered [ 2008.425116] Key type .llcrypt registered [ 2008.528808] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 2009.437255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2009.584964] alg: No test for adler32 (adler32-zlib) [ 2010.601455] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 2010.862611] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 2012.559178] Key type lgssc registered [ 2013.731924] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2017.534795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2017.550372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2018.802537] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2018.824050] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2018.988427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2019.997866] Lustre: Failing over lustre-MDT0000 [ 2020.345687] Lustre: server umount lustre-MDT0000 complete [ 2026.548976] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 18:39:18 (1786401558) [ 2033.219753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2033.746056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2038.626479] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2047.448400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2047.534273] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2047.555970] Lustre: 64678:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2047.834270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2052.541950] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2060.193172] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2065.495979] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2072.988746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2073.186041] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2073.404067] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2080.406345] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2082.944164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2088.792748] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2095.073312] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2103.791427] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2103.798328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2103.815506] Lustre: Skipped 2 previous similar messages [ 2103.842455] Lustre: Skipped 2 previous similar messages [ 2107.692755] Lustre: server umount lustre-MDT0000 complete [ 2111.552170] LustreError: 64668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786401644 with bad export cookie 5254381747188682628 [ 2111.557572] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2111.561932] LustreError: 64668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2111.890395] Lustre: server umount lustre-MDT0001 complete [ 2115.285419] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2119.612684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2128.455636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2130.400288] Lustre: 63598:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786401647/real 1786401647] req@000000006b2eec75 x1873177784393600/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786401663 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2130.434479] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2132.447339] Lustre: 63599:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786401649/real 1786401649] req@0000000003b9a2e4 x1873177784393728/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786401665 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2135.519177] Lustre: 63601:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786401652/real 1786401652] req@0000000087af24a8 x1873177784393856/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786401668 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2136.848271] LustreError: 66526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2136.869646] LustreError: 66526:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2136.967883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2137.075495] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2137.951378] Lustre: 63601:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786401654/real 1786401654] req@00000000f8efd4ff x1873177784394112/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786401670 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2140.004506] LustreError: 66527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2141.036052] LustreError: 66526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2141.050821] LustreError: 66526:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2141.403742] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2142.047385] Lustre: 63599:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786401659/real 1786401659] req@0000000021d07c0a x1873177784394240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786401675 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2146.274404] LustreError: 68450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2146.284765] LustreError: 68450:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2149.636429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2150.009406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2154.856590] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2155.508841] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2159.618577] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2161.588536] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2167.636163] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2174.949456] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2174.956889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2174.968883] Lustre: Skipped 1 previous similar message [ 2180.067237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2180.073457] Lustre: Skipped 4 previous similar messages [ 2180.529774] Lustre: server umount lustre-MDT0000 complete [ 2184.319924] LustreError: 64668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786401717 with bad export cookie 5254381747188684903 [ 2184.326207] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2184.329899] LustreError: 64668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2184.663553] Lustre: server umount lustre-MDT0001 complete [ 2190.878211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2200.597768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2201.567187] Lustre: 63598:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786401718/real 1786401718] req@000000005116aab2 x1873177784421376/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786401734 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2201.612621] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2201.633739] Lustre: Skipped 1 previous similar message [ 2209.951948] LustreError: 63597:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000039b64ba9 x1873177784422400/t0(0) o250->MGC192.168.202.104@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2210.284509] LustreError: 66526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2210.400727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2210.463062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2217.027296] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2222.049489] LustreError: 71133:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2222.067983] LustreError: 71133:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2227.658462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2229.993476] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2232.507268] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2240.448451] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2246.661858] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2253.801439] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2253.808047] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2253.821698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2260.056803] Lustre: server umount lustre-MDT0000 complete [ 2260.965682] LustreError: 71404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2260.993679] LustreError: 71404:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2264.463104] LustreError: 66532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786401797 with bad export cookie 5254381747188686282 [ 2264.474860] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2264.482172] LustreError: 66532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2264.966930] Lustre: server umount lustre-MDT0001 complete [ 2270.766694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2273.868904] Lustre: DEBUG MARKER: files compared the same [ 2286.355775] Lustre: server umount lustre-OST0000 complete [ 2303.989368] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 2306.455520] Key type lgssc unregistered [ 2306.743011] LNet: 74411:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2306.752376] LNetError: 74411:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2306.774287] LNet: Removed LNI 192.168.202.104@tcp [ 2307.563195] Key type .llcrypt unregistered [ 2307.568710] Key type ._llcrypt unregistered [ 2331.979522] Key type ._llcrypt registered [ 2331.981821] Key type .llcrypt registered [ 2332.070582] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 2333.379627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2333.540697] alg: No test for adler32 (adler32-zlib) [ 2334.575224] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 2334.890689] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 2336.618924] Key type lgssc registered [ 2337.631474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2345.608620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2345.626305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2347.134199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2351.559383] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug +malloc all [ 2357.733305] LustreError: 75827:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2357.738436] LustreError: 75827:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2361.182974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2361.489152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2368.072736] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug +malloc all [ 2377.280747] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2384.326516] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2392.244023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2392.492954] LustreError: 77647:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2392.763677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2398.183877] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2398.375559] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug +malloc all [ 2407.023777] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2416.608944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2416.614836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2416.627469] Lustre: Skipped 1 previous similar message [ 2421.730463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2421.733303] Lustre: Skipped 1 previous similar message [ 2422.541774] Lustre: server umount lustre-OST0000 complete [ 2426.848850] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2426.857476] Lustre: Skipped 1 previous similar message [ 2426.858309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2426.864459] Lustre: Skipped 1 previous similar message [ 2431.972194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2431.987502] Lustre: Skipped 2 previous similar messages [ 2432.936483] Lustre: server umount lustre-MDT0000 complete [ 2437.101506] LustreError: 75827:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2437.127588] LustreError: 75827:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2437.137553] LustreError: 75809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786401970 with bad export cookie 11612281243257094194 [ 2437.153653] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2437.592309] Lustre: server umount lustre-MDT0001 complete [ 2450.532498] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 2453.774929] Key type lgssc unregistered [ 2454.177430] LNet: 79363:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2454.181254] LNetError: 79363:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2454.193696] LNet: Removed LNI 192.168.202.104@tcp [ 2455.425151] Key type .llcrypt unregistered [ 2455.428045] Key type ._llcrypt unregistered [ 2479.420522] Key type ._llcrypt registered [ 2479.423416] Key type .llcrypt registered [ 2479.554295] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 2480.585612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2480.781681] alg: No test for adler32 (adler32-zlib) [ 2481.796479] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 2482.088818] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 2483.743846] Key type lgssc registered [ 2485.088821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2494.185905] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2494.201360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2509.855463] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2514.976556] LustreError: 80756:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.104@tcp: failed processing log, type 4: rc = -110 [ 2541.663699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2548.901717] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2556.871239] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2564.909401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2565.018376] Lustre: *** cfs_fail_loc=706, val=0*** [ 2565.477112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2565.550703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2569.369533] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2570.721252] LustreError: 80782:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2570.736317] LustreError: 80782:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2575.843170] LustreError: 80781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2577.526197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2577.941980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2582.843111] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2590.825327] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2597.089619] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2599.395627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2599.409633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2604.530302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2604.548426] Lustre: Skipped 2 previous similar messages [ 2605.488592] Lustre: server umount lustre-OST0000 complete [ 2609.635216] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2609.651360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2609.652742] Lustre: Skipped 2 previous similar messages [ 2609.674138] Lustre: Skipped 1 previous similar message [ 2614.756344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2614.765946] Lustre: Skipped 1 previous similar message [ 2615.954861] Lustre: server umount lustre-MDT0000 complete [ 2619.876260] LustreError: 81845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2619.900331] LustreError: 81845:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2619.970711] LustreError: 80763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786402153 with bad export cookie 459584597293113967 [ 2619.981516] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2619.981893] LustreError: 80763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2620.537919] Lustre: server umount lustre-MDT0001 complete [ 2632.519334] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 2636.057201] Key type lgssc unregistered [ 2636.475349] LNet: 84371:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2636.489274] LNetError: 84371:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2636.503958] LNet: Removed LNI 192.168.202.104@tcp [ 2637.620677] Key type .llcrypt unregistered [ 2637.628652] Key type ._llcrypt unregistered [ 2663.906548] Key type ._llcrypt registered [ 2663.908393] Key type .llcrypt registered [ 2663.984756] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 2665.587896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2665.630774] alg: No test for adler32 (adler32-zlib) [ 2666.646744] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 2666.956293] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 2668.592251] Key type lgssc registered [ 2669.433974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2678.163199] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2678.176667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2685.437597] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2694.725538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2695.498862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2700.633290] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2700.769087] LustreError: 86476:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2700.796922] LustreError: 86476:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2705.888872] LustreError: 86479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2705.895360] LustreError: 86479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2709.523516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2709.744133] LustreError: 86474:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2709.749089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2709.753287] LustreError: 86474:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2714.892729] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2715.105557] LustreError: 86480:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2715.136124] LustreError: 86480:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2720.231162] LustreError: 87224:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2720.257740] LustreError: 87224:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2723.313560] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2723.344211] Lustre: MGS: Client 0f342b82-5c8f-416e-81c0-a2da35438ae1 (at 0@lo) reconnecting [ 2723.364795] LustreError: 86490:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2723.518241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2723.577138] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2727.893623] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2740.704525] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2740.713421] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2740.730070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2744.294876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2744.300378] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2744.302363] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2746.492214] Lustre: server umount lustre-OST0000 complete [ 2754.528665] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2754.535719] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2754.562702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2754.589453] Lustre: Skipped 2 previous similar messages [ 2756.695138] Lustre: server umount lustre-MDT0000 complete [ 2759.648103] LustreError: 86480:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2759.658311] LustreError: 86480:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2761.228174] Lustre: server umount lustre-MDT0001 complete [ 2764.969157] Lustre: server umount MGS complete [ 2775.362432] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 18:51:46 (1786402306) [ 2804.290939] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 2815.767729] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 2856.055319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2861.962686] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2874.515865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2874.744545] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2874.828340] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2874.889583] Lustre: lustre-MDT0001: new disk, initializing [ 2874.961666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2875.009419] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2875.040568] Lustre: 93645:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2875.047360] LustreError: 93645:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2879.401608] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2891.424828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2891.790617] Lustre: lustre-OST0000: new disk, initializing [ 2891.797753] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2891.805869] Lustre: 94412:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2891.900101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2899.364798] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2899.561681] Lustre: 94710:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000987d90c3 x1873178472258688/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786402448 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2909.587372] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2916.187309] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2916.204220] Lustre: MGS: Client d06be130-5cff-43b1-994c-4f187fe87df1 (at 0@lo) reconnecting [ 2916.284299] LustreError: 93631:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2916.327073] Lustre: 92914:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2916.386923] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2916.504551] Lustre: lustre-MDT0000: new disk, initializing [ 2916.721950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2916.806351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2916.841060] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2917.856164] Lustre: 95482:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000083e8361 x1873178472265728/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786402466 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2921.510893] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2923.041813] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2926.957128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2926.964157] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2927.189621] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2952.689967] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2952.701039] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2952.716512] Lustre: Skipped 1 previous similar message [ 2952.727498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2956.848946] Lustre: server umount lustre-OST0000 complete [ 2964.454349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2964.463771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2964.475926] LustreError: Skipped 1 previous similar message [ 2964.508682] Lustre: Skipped 1 previous similar message [ 2964.514298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2964.514307] Lustre: Skipped 3 previous similar messages [ 2966.418777] Lustre: server umount lustre-MDT0000 complete [ 2969.568461] LustreError: 95909:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2969.602681] LustreError: 95909:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2976.357509] Lustre: server umount lustre-MDT0001 complete [ 2980.203640] Lustre: server umount MGS complete [ 2992.271890] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 18:55:23 (1786402523) [ 3012.738792] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 3016.481985] Key type lgssc unregistered [ 3016.937741] LNet: 97981:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3016.949383] LNetError: 97981:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3016.989578] LNet: Removed LNI 192.168.202.104@tcp [ 3017.964858] Key type .llcrypt unregistered [ 3017.968294] Key type ._llcrypt unregistered [ 3035.659128] Key type ._llcrypt registered [ 3035.662260] Key type .llcrypt registered [ 3035.820584] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 3037.303879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3037.355343] alg: No test for adler32 (adler32-zlib) [ 3038.612370] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 3038.830318] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 3040.527185] Key type lgssc registered [ 3041.824272] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3049.291506] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3049.310899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3050.918860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3055.664299] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3060.196669] LustreError: 99201:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3060.217727] LustreError: 99201:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3063.729513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3063.971461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3068.535435] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3076.891337] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3083.830467] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3090.748634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3091.194267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3095.309070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3099.007694] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3108.029911] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3119.074889] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3119.086814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3119.091501] Lustre: Skipped 1 previous similar message [ 3119.114850] Lustre: Skipped 1 previous similar message [ 3123.229538] Lustre: server umount lustre-OST0000 complete [ 3129.325442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3129.341078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3129.345300] Lustre: Skipped 1 previous similar message [ 3129.369026] Lustre: Skipped 1 previous similar message [ 3134.335754] Lustre: server umount lustre-MDT0000 complete [ 3134.444112] LustreError: 101215:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3134.468717] LustreError: 101215:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3137.717961] LustreError: 99181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786402670 with bad export cookie 8028351249659007948 [ 3137.725920] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3137.730939] LustreError: 99181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3138.084711] Lustre: server umount lustre-MDT0001 complete [ 3149.772433] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 3152.808730] Key type lgssc unregistered [ 3153.085079] LNet: 102742:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3153.093831] LNetError: 102742:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3153.121377] LNet: Removed LNI 192.168.202.104@tcp [ 3154.122486] Key type .llcrypt unregistered [ 3154.124624] Key type ._llcrypt unregistered [ 3170.803412] Key type ._llcrypt registered [ 3170.805450] Key type .llcrypt registered [ 3170.877536] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 3171.643830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3171.728939] alg: No test for adler32 (adler32-zlib) [ 3172.691887] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 3172.950167] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 3174.592061] Key type lgssc registered [ 3175.751926] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3183.313312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3183.325817] LustreError: 103815:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3183.493519] LustreError: 103828:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3183.498807] LustreError: 103815:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4838 [ 3183.502647] LustreError: 103815:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3183.526377] LustreError: 103815:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3183.550347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3185.061592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3194.190988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3194.646980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3199.454553] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3206.193493] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing lsmod [ 3209.468321] LustreError: 105036:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3209.481153] LustreError: 105037:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3209.485118] LustreError: 105036:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4988 [ 3209.495196] LustreError: 105036:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3209.508341] LustreError: 105036:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3209.523766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3210.225528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3211.263907] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:65) [ 3215.344686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3215.352753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3215.397783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3216.353110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3216.354806] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3216.363271] Lustre: Skipped 1 previous similar message [ 3216.383863] Lustre: Skipped 1 previous similar message [ 3219.510222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3219.528929] Lustre: Skipped 3 previous similar messages [ 3219.847935] Lustre: server umount lustre-MDT0000 complete [ 3224.501145] LustreError: 103842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786402757 with bad export cookie 1176180167209077925 [ 3224.513754] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3224.522684] LustreError: 103842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3224.555060] LustreError: 103861:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3224.555477] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3224.566597] LustreError: 103861:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3224.602578] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3225.759628] LustreError: 104297:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e62d7570 x1873179002820352/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3226.059771] Lustre: server umount lustre-MDT0001 complete [ 3230.665642] Lustre: server umount lustre-OST0000 complete [ 3239.904111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3240.406552] LustreError: 106099:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3240.521790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3245.552901] LustreError: 106100:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3245.605674] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3250.657387] LustreError: 106099:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3255.063352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3255.683140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3261.194657] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3270.467443] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3277.951970] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3285.858471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3286.366549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3292.977678] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3301.589239] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3310.979755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:97) [ 3321.315772] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3321.317808] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3321.335910] Lustre: Skipped 1 previous similar message [ 3321.348590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3321.349425] LustreError: Skipped 1 previous similar message [ 3324.718142] Lustre: server umount lustre-OST0000 complete [ 3331.554142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3331.563859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3331.570423] Lustre: Skipped 1 previous similar message [ 3331.594017] Lustre: Skipped 2 previous similar messages [ 3335.984394] Lustre: server umount lustre-MDT0000 complete [ 3336.673733] LustreError: 107064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3336.708398] LustreError: 107064:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3340.207953] LustreError: 106081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786402873 with bad export cookie 1176180167209078856 [ 3340.230502] LustreError: 106081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3340.232061] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3340.690536] Lustre: server umount lustre-MDT0001 complete [ 3353.950205] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 3356.643230] Key type lgssc unregistered [ 3357.166362] LNet: 109668:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3357.173392] LNetError: 109668:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3357.200230] LNet: Removed LNI 192.168.202.104@tcp [ 3358.353636] Key type .llcrypt unregistered [ 3358.360279] Key type ._llcrypt unregistered [ 3388.102844] Key type ._llcrypt registered [ 3388.108040] Key type .llcrypt registered [ 3388.255350] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 3389.868484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3389.911391] alg: No test for adler32 (adler32-zlib) [ 3391.145233] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 3391.454593] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 3393.223210] Key type lgssc registered [ 3394.966961] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3403.036078] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3403.060925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3404.686301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3410.745376] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3413.995765] LustreError: 111085:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3414.005362] LustreError: 111085:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3419.107108] LustreError: 111086:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3420.066264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3420.501716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3425.721217] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3434.531767] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3441.429982] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3448.216439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3448.531692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3449.622726] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3456.395178] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3464.743277] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3471.141539] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3475.657410] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3485.664858] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3485.669065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3485.692664] Lustre: Skipped 1 previous similar message [ 3485.705902] Lustre: Skipped 1 previous similar message [ 3487.473263] Lustre: server umount lustre-OST0000 complete [ 3492.320294] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3492.325464] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3492.336802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3495.904725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3495.906327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3495.920014] Lustre: Skipped 1 previous similar message [ 3497.938398] Lustre: server umount lustre-MDT0000 complete [ 3501.034619] LustreError: 111085:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3502.757267] LustreError: 113619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786403035 with bad export cookie 2500268956486039036 [ 3502.759455] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3502.776951] LustreError: 113619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3503.446661] Lustre: server umount lustre-MDT0001 complete [ 3518.768866] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 3523.913126] Key type lgssc unregistered [ 3524.316703] LNet: 114727:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3524.320908] LNetError: 114727:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3524.336092] LNet: Removed LNI 192.168.202.104@tcp [ 3525.005816] Key type .llcrypt unregistered [ 3525.007742] Key type ._llcrypt unregistered [ 3545.492119] Key type ._llcrypt registered [ 3545.494137] Key type .llcrypt registered [ 3545.750694] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 3547.288189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3547.511480] alg: No test for adler32 (adler32-zlib) [ 3549.125506] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 3549.450752] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 3551.339693] Key type lgssc registered [ 3553.897595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3563.734583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3563.752838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3565.275726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3570.581958] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3575.779762] LustreError: 115950:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3575.809437] LustreError: 115950:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3580.898791] LustreError: 115951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3581.048195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3581.553090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3587.031969] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3594.871561] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3601.451925] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3609.023967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3609.656311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3610.727739] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 3617.381887] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3625.472427] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3636.716128] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3636.727158] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3636.731955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3641.111220] Lustre: server umount lustre-OST0000 complete [ 3646.945313] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3646.957379] Lustre: Skipped 1 previous similar message [ 3646.962510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3646.973756] Lustre: Skipped 1 previous similar message [ 3651.046523] Lustre: server umount lustre-MDT0000 complete [ 3652.067644] LustreError: 116933:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3652.096547] LustreError: 116933:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3655.184883] LustreError: 115932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786403188 with bad export cookie 14108340966904016074 [ 3655.199899] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3655.204683] LustreError: 115932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3655.611633] Lustre: server umount lustre-MDT0001 complete [ 3668.040930] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 3671.321771] Key type lgssc unregistered [ 3671.631927] LNet: 119485:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3671.636490] LNetError: 119485:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3671.649477] LNet: Removed LNI 192.168.202.104@tcp [ 3672.864884] Key type .llcrypt unregistered [ 3672.867825] Key type ._llcrypt unregistered [ 3699.765798] Key type ._llcrypt registered [ 3699.768646] Key type .llcrypt registered [ 3699.924250] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 3701.339253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3701.475553] alg: No test for adler32 (adler32-zlib) [ 3702.680884] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 3702.938744] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 3704.631184] Key type lgssc registered [ 3705.618735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3713.831911] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3713.866400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3715.673349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3720.848319] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3725.795938] LustreError: 120901:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3725.803607] LustreError: 120901:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3729.997767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3730.402646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3736.399846] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3746.650707] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3753.214216] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3761.725107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3762.080317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3768.971789] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3776.731464] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3785.679841] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 3795.268918] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3800.621644] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3800.636983] Lustre: Skipped 2 previous similar messages [ 3803.778589] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3803.814575] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3805.455732] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3805.459732] Lustre: Skipped 2 previous similar messages [ 3813.444493] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3813.450211] Lustre: Skipped 1 previous similar message [ 3820.291773] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.4@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3820.301383] Lustre: Skipped 2 previous similar messages [ 3829.862800] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.4@tcp 192.168.0.[2,10]@tcp [ 3834.709357] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.4/24@tcp in log lustre-MDT0000 [ 3834.719941] Lustre: Skipped 2 previous similar messages [ 3839.592238] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.4/24@tcp [ 3839.599435] Lustre: Skipped 1 previous similar message [ 3847.136661] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3847.144815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3847.161183] Lustre: Skipped 1 previous similar message [ 3852.279456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3852.289175] Lustre: Skipped 2 previous similar messages [ 3853.169990] Lustre: server umount lustre-OST0000 complete [ 3857.376902] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3857.385714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3857.404338] Lustre: Skipped 1 previous similar message [ 3857.424967] Lustre: Skipped 1 previous similar message [ 3862.498745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3862.514171] Lustre: Skipped 1 previous similar message [ 3863.737372] Lustre: server umount lustre-MDT0000 complete [ 3867.616283] LustreError: 120902:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3867.860073] LustreError: 120883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786403400 with bad export cookie 15628392408753673255 [ 3867.871928] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3867.888690] LustreError: 120883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3868.270661] Lustre: server umount lustre-MDT0001 complete [ 3880.029209] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 3882.635621] Key type lgssc unregistered [ 3882.960758] LNet: 126063:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3882.969143] LNetError: 126063:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3882.984426] LNet: Removed LNI 192.168.202.104@tcp [ 3883.998926] Key type .llcrypt unregistered [ 3884.000368] Key type ._llcrypt unregistered [ 3914.485226] Key type ._llcrypt registered [ 3914.489237] Key type .llcrypt registered [ 3914.645336] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 3915.671175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3915.703637] alg: No test for adler32 (adler32-zlib) [ 3916.828560] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 3917.243291] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 3918.943887] Key type lgssc registered [ 3920.428917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3929.387320] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3929.400879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3930.921572] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3930.933656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3930.973220] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.4/24@tcp [ 3936.825878] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3940.328208] LustreError: 127626:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3940.355535] LustreError: 127626:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3944.634393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3944.898736] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3944.935201] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.4/24@tcp [ 3944.965664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3949.488718] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3956.391707] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3961.520410] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3968.517814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3968.978251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3975.158167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:201 to 0x280000401:225) [ 3975.392199] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3983.462424] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3995.628571] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3995.629541] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3995.648510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3995.653584] Lustre: Skipped 1 previous similar message [ 3999.348854] Lustre: server umount lustre-OST0000 complete [ 4005.857023] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4005.876730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4005.881047] Lustre: Skipped 1 previous similar message [ 4005.903507] Lustre: Skipped 2 previous similar messages [ 4009.016155] Lustre: server umount lustre-MDT0000 complete [ 4010.982341] LustreError: 128874:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4011.006780] LustreError: 128874:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4013.394201] LustreError: 127607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786403546 with bad export cookie 2206825212072519023 [ 4013.406107] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4013.413229] LustreError: 127607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4013.966527] Lustre: server umount lustre-MDT0001 complete [ 4026.957220] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 4030.637301] Key type lgssc unregistered [ 4030.979787] LNet: 131215:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4030.996737] LNetError: 131215:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4031.017286] LNet: Removed LNI 192.168.202.104@tcp [ 4032.167569] Key type .llcrypt unregistered [ 4032.172737] Key type ._llcrypt unregistered [ 4060.361638] Key type ._llcrypt registered [ 4060.365171] Key type .llcrypt registered [ 4060.513895] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 4077.542833] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 4079.404756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4079.453829] alg: No test for adler32 (adler32-zlib) [ 4080.516384] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 4080.827779] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 4082.535597] Key type lgssc registered [ 4084.174319] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4123.644653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4123.689470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4124.977493] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4125.008302] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4125.103217] Lustre: lustre-MDT0000: new disk, initializing [ 4125.221649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4125.242538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4130.230165] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4141.555169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4141.816493] Lustre: 134620:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4141.895718] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4141.900262] Lustre: Skipped 1 previous similar message [ 4142.004431] Lustre: lustre-MDT0001: new disk, initializing [ 4142.163255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4142.226243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4142.250938] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4149.222229] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4162.403582] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4167.412832] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4174.841789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4175.282526] Lustre: lustre-OST0000: new disk, initializing [ 4175.285166] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4175.305388] Lustre: 136586:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4175.435159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4182.648204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4182.658182] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4182.854366] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4183.145598] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4193.740342] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4198.780621] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4198.989764] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4203.578549] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4203.811947] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4208.632283] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4208.634721] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4208.637995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4208.656396] LustreError: Skipped 1 previous similar message [ 4208.696543] Lustre: Skipped 1 previous similar message [ 4213.732932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4213.737343] Lustre: Skipped 2 previous similar messages [ 4218.859972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4218.871502] Lustre: Skipped 1 previous similar message [ 4219.872240] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4220.074973] Lustre: server umount lustre-OST0000 complete [ 4229.093820] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4229.110922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4229.118537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4230.372571] Lustre: server umount lustre-MDT0000 complete [ 4234.349886] LustreError: 134612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786403767 with bad export cookie 7374313212448740220 [ 4234.355168] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4234.360477] LustreError: 134612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4234.897479] Lustre: server umount lustre-MDT0001 complete [ 4244.503258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4244.979452] LustreError: 138813:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4245.000168] LustreError: 138813:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4245.141776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4250.358063] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4250.604631] LustreError: 138814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4250.634574] LustreError: 138814:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4255.712857] LustreError: 138813:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4259.980621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4260.451112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4265.771195] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4273.533806] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4279.532769] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4286.430268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4286.823669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4293.477747] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4300.772352] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4306.422825] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4306.773095] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4312.228986] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4312.511916] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4321.825753] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4322.135239] Lustre: lustre-OST0001: new disk, initializing [ 4322.138483] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4322.146289] Lustre: 142187:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4322.256538] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4329.102517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4329.122277] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4329.336889] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4329.374924] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4338.539810] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4338.808113] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4343.623309] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4343.874841] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4351.224323] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9964034b7000.ost_server_uuid 50 [ 4353.205189] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9964034b7000.ost_server_uuid in IDLE state after 0 sec [ 4365.140367] Lustre: server umount lustre-OST0001 complete [ 4365.298399] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4365.304103] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4365.306620] LustreError: Skipped 1 previous similar message [ 4365.312894] LustreError: 140637:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4365.316920] Lustre: Skipped 2 previous similar messages [ 4365.354285] LustreError: 140637:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4370.411323] LustreError: 140638:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4370.416668] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4370.431078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4370.431088] Lustre: Skipped 1 previous similar message [ 4370.436734] LustreError: 140638:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4370.491449] Lustre: Skipped 1 previous similar message [ 4375.996513] Lustre: server umount lustre-OST0000 complete [ 4380.646671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4380.658905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4380.667478] Lustre: Skipped 4 previous similar messages [ 4386.446611] Lustre: server umount lustre-MDT0000 complete [ 4390.705963] LustreError: 138796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786403923 with bad export cookie 7374313212448741102 [ 4390.713692] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4390.716477] LustreError: 138796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4391.019338] Lustre: server umount lustre-MDT0001 complete [ 4411.455765] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 4414.829919] Key type lgssc unregistered [ 4415.171792] LNet: 145190:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4415.188340] LNetError: 145190:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4415.201605] LNet: Removed LNI 192.168.202.104@tcp [ 4416.151540] Key type .llcrypt unregistered [ 4416.163154] Key type ._llcrypt unregistered [ 4457.185346] Key type ._llcrypt registered [ 4457.189450] Key type .llcrypt registered [ 4457.379820] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 4471.625091] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 4473.071220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4473.100107] alg: No test for adler32 (adler32-zlib) [ 4474.349705] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 4474.647403] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 4476.311160] Key type lgssc registered [ 4478.191243] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4521.340156] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4521.357931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4522.650706] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4522.734800] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4522.950449] Lustre: lustre-MDT0000: new disk, initializing [ 4523.148221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4523.242298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4529.257390] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4540.512187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4540.660920] Lustre: 149344:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4540.730965] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4540.738670] Lustre: Skipped 1 previous similar message [ 4540.846298] Lustre: lustre-MDT0001: new disk, initializing [ 4540.960458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4540.994713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4541.006264] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4546.683640] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4557.538522] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4564.411450] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4572.406251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4572.792314] Lustre: lustre-OST0000: new disk, initializing [ 4572.798872] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4572.810568] Lustre: 151314:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4572.944193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4579.917432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4579.926370] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4580.089981] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4580.166981] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4589.836499] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4593.990747] Lustre: Failing over lustre-OST0000 [ 4594.453838] Lustre: server umount lustre-OST0000 complete [ 4611.042337] Lustre: 147567:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786404128/real 1786404128] req@000000005378dddb x1873180368079104/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786404144 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4611.073379] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4612.065277] Lustre: 147568:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786404128/real 1786404128] req@000000005a42b352 x1873180368079232/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1786404144 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4612.108168] Lustre: 147568:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4613.964044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4614.458486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4614.480143] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4616.479281] Lustre: 147570:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786404133/real 1786404133] req@000000000aff8d5a x1873180368081024/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786404149 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4621.505399] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4621.600511] Lustre: 147569:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786404138/real 1786404138] req@00000000e4bdde8b x1873180368082048/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786404154 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4621.611562] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4621.642162] Lustre: 147569:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4622.259707] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4622.265728] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4625.245729] Lustre: Failing over lustre-MDT0000 [ 4625.607483] Lustre: server umount lustre-MDT0000 complete [ 4626.913719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4626.936891] Lustre: Skipped 2 previous similar messages [ 4627.999174] Lustre: 147568:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786404144/real 1786404144] req@00000000d93426e1 x1873180368083200/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786404160 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4628.013661] Lustre: 147568:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4632.034579] LustreError: 149357:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4632.050643] LustreError: 149357:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4637.156345] LustreError: 149356:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4637.181325] LustreError: 149356:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4642.207201] Lustre: 147570:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786404159/real 1786404159] req@000000000aff8d5a x1873180368092416/t0(0) o400->MGC192.168.202.104@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786404175 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4642.242808] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4642.291248] LustreError: 151113:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4642.332747] LustreError: 151113:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4645.838297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4647.392666] LustreError: 149356:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4647.408244] LustreError: 149356:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4652.511911] LustreError: 147566:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000016ef8768 x1873180368097152/t0(0) o250->MGC192.168.202.104@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4652.550855] LustreError: 149351:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4652.573728] LustreError: 149351:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4652.978333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4653.079517] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4653.876161] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4658.173515] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4658.190741] Lustre: Skipped 1 previous similar message [ 4658.217779] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4658.264118] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4658.279169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4668.384029] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4668.395723] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4668.409333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4668.424157] Lustre: Skipped 2 previous similar messages [ 4668.459685] Lustre: Skipped 1 previous similar message [ 4673.501878] Lustre: server umount lustre-OST0000 complete [ 4678.628027] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4678.637101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4678.637712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4678.664600] Lustre: Skipped 1 previous similar message [ 4683.745061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4683.754358] Lustre: Skipped 2 previous similar messages [ 4684.622580] Lustre: server umount lustre-MDT0000 complete [ 4688.870545] LustreError: 151112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4688.885553] LustreError: 151112:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4689.409498] LustreError: 149335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786404222 with bad export cookie 14487041204931569865 [ 4689.422812] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4689.429063] LustreError: 149335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4689.656381] Lustre: server umount lustre-MDT0001 complete [ 4701.938820] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 4705.283373] Key type lgssc unregistered [ 4705.500529] LNet: 154973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4705.507678] LNetError: 154973:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4705.527344] LNet: Removed LNI 192.168.202.104@tcp [ 4706.588991] Key type .llcrypt unregistered [ 4706.592590] Key type ._llcrypt unregistered [ 4746.852910] Key type ._llcrypt registered [ 4746.855830] Key type .llcrypt registered [ 4746.992315] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 4748.842545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4749.322379] alg: No test for adler32 (adler32-zlib) [ 4750.486875] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 4750.750385] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 4752.463943] Key type lgssc registered [ 4753.907247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4761.283821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4761.306082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4762.758242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4767.552444] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4772.325753] LustreError: 156494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4772.359208] LustreError: 156494:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4777.447082] LustreError: 156495:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4777.699898] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4781.746798] LustreError: 156523:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4781.758269] LustreError: 156523:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4781.771991] LustreError: 156523:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4782.145946] Lustre: server umount lustre-MDT0000 complete [ 4799.529743] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 4811.349568] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 4846.246892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4846.508558] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4846.533849] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4846.619644] Lustre: lustre-MDT0000: new disk, initializing [ 4846.761297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4846.794686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4852.260076] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4864.641411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4864.725886] Lustre: 159829:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4864.763954] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4864.766937] Lustre: Skipped 1 previous similar message [ 4864.818253] Lustre: lustre-MDT0001: new disk, initializing [ 4864.893451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4864.919605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4864.928765] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4871.877148] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4883.333880] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4890.110639] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4897.329293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4897.792226] Lustre: lustre-OST0000: new disk, initializing [ 4897.798863] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4897.806466] Lustre: 161798:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4897.936294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4903.504747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4903.517940] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4903.676782] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4904.212573] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4915.438660] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4919.298759] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4921.333611] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 19:27:32 (1786404452) [ 5156.845267] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5156.861621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5156.878800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5159.905841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5159.912324] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5159.932270] Lustre: Skipped 1 previous similar message [ 5161.844165] Lustre: server umount lustre-MDT0000 complete [ 5165.024627] LustreError: 161440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5165.053541] LustreError: 161440:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5165.932816] LustreError: 159819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786404698 with bad export cookie 11806604521263762203 [ 5165.934960] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5165.946361] LustreError: 159819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5170.145644] LustreError: 159836:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5170.146174] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5170.182753] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5170.187943] Lustre: Skipped 2 previous similar messages [ 5175.263924] LustreError: 163585:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5175.264399] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5180.383141] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5180.619905] Lustre: server umount lustre-MDT0001 complete [ 5190.548041] Lustre: server umount lustre-OST0000 complete [ 5198.440635] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 5208.814226] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 5241.605838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5241.843436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5241.873907] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5241.966345] Lustre: lustre-MDT0000: new disk, initializing [ 5242.055342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5242.076675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5248.438026] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5259.585815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5259.729989] Lustre: 166459:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5259.820902] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5259.831631] Lustre: Skipped 1 previous similar message [ 5259.952665] Lustre: lustre-MDT0001: new disk, initializing [ 5260.063601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5260.099390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5260.118265] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5265.045428] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5274.988584] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5280.352674] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5287.139665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5287.545919] Lustre: lustre-OST0000: new disk, initializing [ 5287.551844] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5287.566431] Lustre: 168427:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5287.674131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5293.945740] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5297.749073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5297.761093] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5297.889091] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5303.323867] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5308.105817] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5308.400118] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5313.133463] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5313.389637] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5318.625163] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5318.625227] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5318.632664] LustreError: Skipped 1 previous similar message [ 5318.663725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5318.663734] Lustre: Skipped 1 previous similar message [ 5321.725973] Lustre: server umount lustre-OST0000 complete [ 5326.818628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5326.829069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5326.838832] Lustre: Skipped 1 previous similar message [ 5326.853740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5326.863959] Lustre: Skipped 1 previous similar message [ 5331.613719] Lustre: server umount lustre-MDT0000 complete [ 5333.989980] LustreError: 166470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5334.004393] LustreError: 166470:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5335.675444] LustreError: 166451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786404868 with bad export cookie 11806604521263921033 [ 5335.691065] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5335.701835] LustreError: 166451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5336.221671] Lustre: server umount lustre-MDT0001 complete [ 5345.788500] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 19:34:36 (1786404876) [ 5364.592851] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 5376.165471] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 5411.451584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5411.722958] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5411.752493] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5411.794187] Lustre: lustre-MDT0000: new disk, initializing [ 5411.833856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5411.846243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5417.090664] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5428.617776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5428.757513] Lustre: 173186:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5429.142037] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5434.815701] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5444.627090] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5449.690833] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5457.505767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5457.970574] Lustre: lustre-OST0000: new disk, initializing [ 5457.973559] Lustre: Skipped 1 previous similar message [ 5457.976802] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5457.981446] Lustre: Skipped 2 previous similar messages [ 5457.987562] Lustre: 175166:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5464.632121] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5467.752480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5467.763038] Lustre: Skipped 1 previous similar message [ 5467.768495] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5467.948039] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5476.240119] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5488.609181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5488.614102] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5488.633619] Lustre: Skipped 1 previous similar message [ 5488.640481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5488.640490] Lustre: Skipped 2 previous similar messages [ 5491.398470] Lustre: server umount lustre-OST0000 complete [ 5495.785179] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5510.111163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5510.312522] Lustre: server umount lustre-MDT0000 complete [ 5513.185624] LustreError: 174358:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5513.194451] LustreError: 174358:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5513.255479] LustreError: 173176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786405046 with bad export cookie 11806604521263921915 [ 5513.261078] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5513.268024] LustreError: 173176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5513.515727] Lustre: server umount lustre-MDT0001 complete [ 5521.960526] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 19:37:33 (1786405053) [ 5537.873091] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 5548.492664] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 5586.956059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5587.227917] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5587.259255] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5587.342480] Lustre: lustre-MDT0000: new disk, initializing [ 5587.415344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5587.422574] Lustre: Skipped 2 previous similar messages [ 5587.434229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5591.004747] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5601.813866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5601.933120] Lustre: 179488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5602.305270] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5606.777925] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5615.222326] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5621.004592] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5628.944545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5629.366620] Lustre: 181465:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5635.637459] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5635.786104] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5636.534701] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5645.829893] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5656.545670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5656.565291] Lustre: Skipped 2 previous similar messages [ 5656.575311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5656.576758] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5656.579974] Lustre: Skipped 9 previous similar messages [ 5657.778121] Lustre: server umount lustre-OST0000 complete [ 5667.370260] Lustre: server umount lustre-MDT0000 complete [ 5670.621472] LustreError: 179481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786405203 with bad export cookie 11806604521263923000 [ 5670.625931] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5670.627702] LustreError: 179481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5670.854748] Lustre: server umount lustre-MDT0001 complete [ 5682.074669] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 5684.705775] Key type lgssc unregistered [ 5685.006805] LNet: 183320:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5685.015405] LNetError: 183320:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5685.029710] LNet: Removed LNI 192.168.202.104@tcp [ 5685.838246] Key type .llcrypt unregistered [ 5685.840090] Key type ._llcrypt unregistered [ 5710.793816] Key type ._llcrypt registered [ 5710.798071] Key type .llcrypt registered [ 5710.872512] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 5712.372993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5712.483126] alg: No test for adler32 (adler32-zlib) [ 5713.764840] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 5714.195523] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 5715.895179] Key type lgssc registered [ 5717.203420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5725.265696] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5725.286271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5726.828322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5731.588952] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5736.932630] LustreError: 184737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5736.954319] LustreError: 184737:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5740.089199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5740.400114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5744.653942] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5752.274498] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5758.341673] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5765.048569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5765.367410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5771.273367] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5771.509649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5778.255529] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5802.471230] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5802.484417] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5802.505489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5806.560099] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5806.562379] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5806.575460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5806.610057] Lustre: Skipped 1 previous similar message [ 5808.082523] Lustre: server umount lustre-OST0000 complete [ 5812.193126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5812.207127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5812.222559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5816.801524] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5816.802593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5816.819996] Lustre: Skipped 1 previous similar message [ 5818.471550] Lustre: server umount lustre-MDT0000 complete [ 5821.930385] LustreError: 184732:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5821.960483] LustreError: 184732:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5822.488587] LustreError: 184716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786405355 with bad export cookie 331666869765220975 [ 5822.508697] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5822.521759] LustreError: 184716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5822.964208] Lustre: server umount lustre-MDT0001 complete [ 5835.069616] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 5838.214422] Key type lgssc unregistered [ 5838.559819] LNet: 188292:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5838.573976] LNetError: 188292:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5838.596420] LNet: Removed LNI 192.168.202.104@tcp [ 5839.464966] Key type .llcrypt unregistered [ 5839.467803] Key type ._llcrypt unregistered [ 5867.808748] Key type ._llcrypt registered [ 5867.821033] Key type .llcrypt registered [ 5867.933543] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 5869.157034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5869.457496] alg: No test for adler32 (adler32-zlib) [ 5870.696307] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 5871.010356] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 5872.751435] Key type lgssc registered [ 5874.732096] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5883.838781] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5883.854174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5885.353943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5891.426212] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5895.649991] LustreError: 189711:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5895.677083] LustreError: 189711:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5900.780734] LustreError: 189712:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5901.227897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5901.566628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5907.075432] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5917.146975] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5924.025269] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5932.720933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5933.163577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5940.150239] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5947.665914] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5955.711255] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5964.687432] Lustre: lustre-OST0000: Not available for connect from 192.168.202.4@tcp (stopping) [ 5966.306345] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5966.314819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5966.316400] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5966.329445] Lustre: Skipped 1 previous similar message [ 5966.373735] Lustre: Skipped 1 previous similar message [ 5969.787524] Lustre: lustre-OST0000: Not available for connect from 192.168.202.4@tcp (stopping) [ 5970.273641] Lustre: server umount lustre-OST0000 complete [ 5979.071857] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9964080da800.ost_server_uuid 50 [ 5980.822844] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5997.024268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5997.027235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5997.045493] Lustre: Skipped 1 previous similar message [ 5997.059539] Lustre: Skipped 1 previous similar message [ 6000.717205] Lustre: server umount lustre-MDT0000 complete [ 6002.147670] LustreError: 189712:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6002.165806] LustreError: 189712:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6005.066203] LustreError: 189693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786405538 with bad export cookie 14463498795043305971 [ 6005.085196] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6005.421807] Lustre: server umount lustre-MDT0001 complete [ 6014.666643] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 19:45:46 (1786405546) [ 6021.473992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6022.161926] LustreError: 193650:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6022.222268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6027.252685] LustreError: 193651:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6028.384414] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6038.500086] LustreError: 193650:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6038.523935] LustreError: 193650:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6039.463094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6039.984823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6044.629512] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6052.817942] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6058.910485] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6066.341028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6066.757385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6068.793891] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6074.213131] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6083.320645] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6092.481424] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6092.822826] Lustre: lustre-OST0001: new disk, initializing [ 6092.827117] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6092.836292] Lustre: 196491:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6092.997503] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6099.784801] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6100.095871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6100.113316] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6100.213917] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6111.001971] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6114.785160] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6114.793681] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6114.811055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6114.827847] Lustre: Skipped 1 previous similar message [ 6120.194944] Lustre: server umount lustre-OST0000 complete [ 6120.951663] LustreError: 195472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6120.970589] LustreError: 195472:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6127.581638] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6127.825511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6133.285465] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6133.498441] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6151.650716] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6151.661992] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6151.662259] LustreError: Skipped 2 previous similar messages [ 6151.669254] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6151.676326] Lustre: Skipped 2 previous similar messages [ 6151.697429] Lustre: Skipped 4 previous similar messages [ 6153.518553] Lustre: server umount lustre-OST0001 complete [ 6157.794666] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6157.804205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6164.081146] Lustre: server umount lustre-MDT0000 complete [ 6167.008677] LustreError: 193645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6167.034704] LustreError: 193645:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 6168.070284] LustreError: 193631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786405701 with bad export cookie 14463498795043307035 [ 6168.076190] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6168.081578] LustreError: 193631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6168.734383] Lustre: server umount lustre-MDT0001 complete [ 6194.597760] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 19:48:45 (1786405725) [ 6202.500154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6202.925422] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6202.999484] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6203.320680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6208.754198] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6218.045871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6218.202239] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6218.263394] Lustre: 199932:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6223.031769] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6230.807396] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6236.964742] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6243.361829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6243.535647] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6251.202340] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6251.922925] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6261.177330] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6269.772000] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6269.850751] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6270.274870] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6270.277962] Lustre: Skipped 2 previous similar messages [ 6278.897918] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6286.999921] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6291.946804] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6291.950346] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6291.951678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6291.951685] Lustre: Skipped 3 previous similar messages [ 6291.996241] Lustre: Skipped 2 previous similar messages [ 6295.457498] Lustre: server umount lustre-OST0000 complete [ 6297.066866] LustreError: 201782:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6297.081325] LustreError: 201782:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6317.541665] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6317.547707] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6317.558011] LustreError: Skipped 1 previous similar message [ 6317.603342] Lustre: Skipped 1 previous similar message [ 6320.326223] Lustre: server umount lustre-OST0001 complete [ 6327.798705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6327.818199] Lustre: Skipped 5 previous similar messages [ 6332.690895] Lustre: server umount lustre-MDT0000 complete [ 6337.452379] LustreError: 199926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786405870 with bad export cookie 14463498795043308449 [ 6337.453305] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6337.477318] LustreError: 199926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6337.817336] Lustre: server umount lustre-MDT0001 complete [ 6364.658783] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 19:51:36 (1786405896) [ 6384.232928] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 6397.021888] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 6439.337401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6439.681967] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6439.744576] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6439.844829] Lustre: lustre-MDT0000: new disk, initializing [ 6439.901538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6439.916312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6446.479053] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6459.609266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6459.800238] Lustre: 208036:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6459.860544] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6459.869436] Lustre: Skipped 1 previous similar message [ 6459.980570] Lustre: lustre-MDT0001: new disk, initializing [ 6460.149070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6460.169611] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6466.036119] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6476.841635] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6483.282508] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6493.754818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6494.369395] Lustre: lustre-OST0000: new disk, initializing [ 6494.382811] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6494.399153] Lustre: 210005:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6503.599498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6503.625680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6503.857699] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6504.714514] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6517.006851] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6521.906163] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6522.102508] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6526.775172] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6527.004425] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6529.504579] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6529.519505] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6529.528194] Lustre: Skipped 2 previous similar messages [ 6529.539434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6529.547814] Lustre: Skipped 1 previous similar message [ 6543.327653] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6543.464458] Lustre: server umount lustre-OST0000 complete [ 6553.713288] Lustre: server umount lustre-MDT0000 complete [ 6555.104792] LustreError: 208047:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6555.137030] LustreError: 208047:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 16 previous similar messages [ 6557.684128] LustreError: 208026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786406090 with bad export cookie 14463498795043310010 [ 6557.686208] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6557.693507] LustreError: 208026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6558.068451] Lustre: server umount lustre-MDT0001 complete [ 6566.834919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6572.144891] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6581.870755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6587.772031] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6596.066796] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6602.751145] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6612.015632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6620.543664] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6628.914294] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6634.651852] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6634.948408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6639.904511] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6640.280615] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6643.176965] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6643.178682] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6643.199028] Lustre: Skipped 3 previous similar messages [ 6643.203551] LustreError: Skipped 2 previous similar messages [ 6648.915964] Lustre: server umount lustre-OST0000 complete [ 6656.414961] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6661.939468] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6693.184820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6698.794517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6698.804403] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6698.804832] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6701.202240] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6709.495636] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6711.515668] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6719.463663] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6719.476149] LustreError: Skipped 1 previous similar message [ 6719.479935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6719.484207] Lustre: Skipped 11 previous similar messages [ 6722.019710] Lustre: server umount lustre-OST0000 complete [ 6732.389215] Lustre: server umount lustre-MDT0000 complete [ 6737.472553] LustreError: 217320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786406270 with bad export cookie 14463498795043310892 [ 6737.482145] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6737.499517] LustreError: 217320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6738.318630] Lustre: server umount lustre-MDT0001 complete [ 6751.315391] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 19:58:01 (1786406281) [ 6759.104689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6759.628845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6759.643201] Lustre: Skipped 6 previous similar messages [ 6763.274266] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6772.925069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6778.183511] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6786.560814] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6793.300243] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6801.296644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6810.037028] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6818.493625] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6824.620117] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6824.829842] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6829.920699] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6830.215764] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6837.820484] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6837.969705] Lustre: lustre-OST0001: new disk, initializing [ 6837.980662] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6838.000680] Lustre: 221564:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6845.726718] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6847.123838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6847.137547] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6847.235237] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6857.178178] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6862.877516] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6863.188601] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6870.280710] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6870.698934] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6878.180064] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6878.204145] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6878.239653] Lustre: Skipped 5 previous similar messages [ 6879.670159] Lustre: server umount lustre-OST0001 complete [ 6883.324660] LustreError: 220015:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6883.365068] LustreError: 220015:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 6887.082257] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6887.308923] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6893.062065] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6893.444963] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6922.990856] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6928.381656] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6928.405811] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6928.406937] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6928.444713] Lustre: Skipped 1 previous similar message [ 6929.698812] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6937.822715] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6939.519286] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6947.512251] Lustre: server umount lustre-OST0001 complete [ 6971.391316] Lustre: server umount lustre-OST0000 complete [ 6978.023771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6978.032083] Lustre: Skipped 11 previous similar messages [ 6981.965559] Lustre: server umount lustre-MDT0000 complete [ 6988.763353] LustreError: 218176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786406521 with bad export cookie 14463498795043311851 [ 6988.765971] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6988.772895] LustreError: 218176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7003.103140] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7003.739291] Lustre: server umount lustre-MDT0001 complete [ 7035.207427] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 20:02:45 (1786406565) [ 7042.730906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7043.410483] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7043.478081] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7050.653899] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7062.031456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7062.165543] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7062.252547] Lustre: 226823:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7068.718780] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7077.722831] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7085.076819] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7094.569881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7095.015986] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7103.705627] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7112.635977] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7124.093468] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7124.170973] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7132.375545] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7140.871408] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7143.486801] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7150.897721] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7151.220359] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7157.617049] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7157.960332] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7163.408505] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9964115ed000.ost_server_uuid 50 [ 7164.773773] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9964115ed000.ost_server_uuid in IDLE state after 0 sec [ 7166.463193] Lustre: Permanently deactivating lustre-OST0001 [ 7166.472119] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7169.245886] Lustre: Permanently reactivating lustre-OST0001 [ 7175.136164] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7175.161325] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7175.162888] LustreError: Skipped 7 previous similar messages [ 7175.215856] Lustre: Skipped 8 previous similar messages [ 7180.134377] Lustre: server umount lustre-OST0001 complete [ 7191.195132] Lustre: server umount lustre-OST0000 complete [ 7202.897548] Lustre: server umount lustre-MDT0000 complete [ 7207.710894] LustreError: 226814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786406740 with bad export cookie 14463498795043313146 [ 7207.713952] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7207.728202] LustreError: 226814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7208.444736] Lustre: server umount lustre-MDT0001 complete [ 7239.119883] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 20:06:10 (1786406770) [ 7249.229259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7249.613808] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7249.656524] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7249.666589] Lustre: Skipped 5 previous similar messages [ 7255.610870] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7266.264238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7266.328316] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7266.351956] Lustre: 233520:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7272.120681] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7280.593654] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7287.117504] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7295.857126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7296.145871] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7296.192390] Lustre: Permanently deactivating lustre-OST0000 [ 7296.208510] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7296.422249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7296.428467] Lustre: Skipped 10 previous similar messages [ 7302.194466] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7304.202948] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7313.324665] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7320.400720] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7320.544076] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7326.414330] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7328.106826] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7336.323073] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7338.250417] Lustre: Permanently reactivating lustre-OST0000 [ 7338.289480] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7338.296567] Lustre: Skipped 2 previous similar messages [ 7346.740864] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7346.740968] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 44 seconds. I think it's dead, and I am evicting it. exp 0000000074bc803a, cur 1786406879 deadline 1786406865 last 1786406835 [ 7346.744815] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7346.777858] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7346.847517] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7353.221125] Lustre: server umount lustre-OST0001 complete [ 7363.820348] Lustre: server umount lustre-OST0000 complete [ 7374.914933] Lustre: server umount lustre-MDT0000 complete [ 7378.895580] LustreError: 237117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786406911 with bad export cookie 14463498795043314791 [ 7378.905165] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7378.916572] LustreError: 237117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7379.441949] Lustre: server umount lustre-MDT0001 complete [ 7392.064266] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 7395.127604] Key type lgssc unregistered [ 7395.575768] LNet: 238400:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7395.581183] LNetError: 238400:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7396.653544] LNet: Removed LNI 192.168.202.104@tcp [ 7398.539870] Key type .llcrypt unregistered [ 7398.550241] Key type ._llcrypt unregistered [ 7428.442219] Key type ._llcrypt registered [ 7428.444761] Key type .llcrypt registered [ 7428.533139] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 7429.424672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7429.462357] alg: No test for adler32 (adler32-zlib) [ 7430.530866] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 7430.834937] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 7432.519156] Key type lgssc registered [ 7433.704225] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7444.971783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7445.004616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7446.768280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7452.007592] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7456.234660] LustreError: 239869:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7456.250929] LustreError: 239869:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7461.001763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7461.179838] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7461.206445] Lustre: 239856:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7461.214161] Lustre: Permanently deactivating lustre-MDT0001 [ 7461.246095] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7461.344765] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7461.640795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7463.104245] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7463.117116] LustreError: 239899:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7463.143522] LustreError: 239899:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7463.161902] LustreError: 239899:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 1, failed: rc = -108 [ 7467.940705] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7476.729338] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7483.443954] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7491.253818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7491.547283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7492.594947] LustreError: 241711:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7492.605483] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7497.704678] LustreError: 241711:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7498.922172] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7502.821361] LustreError: 241781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7506.918931] LustreError: 241781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7506.937470] LustreError: 241781:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7507.526542] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7515.074274] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7515.363808] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7520.764647] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7520.765361] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7522.059146] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7530.480814] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7534.235419] Lustre: Permanently reactivating lustre-MDT0001 [ 7534.252821] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7534.257490] Lustre: Skipped 2 previous similar messages [ 7539.115240] Lustre: lustre-MDT0001: Client b38d7810-5402-4b67-812c-7e325b570daa (at 192.168.202.4@tcp) reconnecting [ 7546.804929] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7548.744710] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7554.729593] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7554.962325] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7559.869878] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7560.110568] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7562.684484] Lustre: Permanently deactivating lustre-MDT0001 [ 7562.700701] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7562.712357] Lustre: Skipped 2 previous similar messages [ 7569.017960] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7582.179175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7582.181379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7582.193537] Lustre: Skipped 3 previous similar messages [ 7582.217662] Lustre: Skipped 3 previous similar messages [ 7586.558879] Lustre: server umount lustre-MDT0000 complete [ 7587.300086] LustreError: 239869:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7587.338528] LustreError: 239869:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7590.640490] LustreError: 239848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786407123 with bad export cookie 13329342567633652556 [ 7590.643947] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7590.653616] LustreError: 239848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7590.956410] Lustre: server umount lustre-MDT0001 complete [ 7596.988857] Lustre: server umount lustre-OST0000 complete [ 7607.770586] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 20:12:19 (1786407139) [ 7618.149992] Lustre: server umount lustre-OST0001 complete [ 7625.528205] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 7638.565281] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 7680.634105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7680.892303] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7680.902197] Lustre: Skipped 2 previous similar messages [ 7680.923963] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7681.014424] Lustre: lustre-MDT0000: new disk, initializing [ 7681.116835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7681.143669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7686.468360] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7697.982089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7698.119729] Lustre: 248147:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7698.154531] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7698.157479] Lustre: Skipped 1 previous similar message [ 7698.220638] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7698.271213] Lustre: lustre-MDT0001: new disk, initializing [ 7698.320567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7698.337816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7698.345537] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7703.759604] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7715.665582] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7723.766984] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7732.437795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7732.864272] Lustre: lustre-OST0000: new disk, initializing [ 7732.867792] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7732.873277] Lustre: 250117:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7732.971372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7740.012490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7740.023606] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7740.144826] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7741.025488] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7751.892913] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7753.877842] LustreError: 248155:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7763.439641] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7763.785917] Lustre: lustre-OST0001: new disk, initializing [ 7763.792704] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7763.806708] Lustre: 251233:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7763.959599] LustreError: 248155:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7764.002613] LustreError: 250365:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7770.907969] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7773.804201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7773.819145] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7773.945370] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7774.083306] LustreError: 250365:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7781.148975] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7783.070513] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7789.543971] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7789.555515] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7789.560704] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7789.566245] Lustre: Skipped 1 previous similar message [ 7789.615448] Lustre: Skipped 1 previous similar message [ 7792.159119] Lustre: server umount lustre-OST0001 complete [ 7794.681925] LustreError: 250102:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7794.719602] LustreError: 250102:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7812.065034] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7812.073927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7812.078939] LustreError: Skipped 1 previous similar message [ 7812.099486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7817.021829] Lustre: server umount lustre-OST0000 complete [ 7825.376795] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7825.378453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7825.407470] Lustre: Skipped 2 previous similar messages [ 7825.431493] Lustre: Skipped 4 previous similar messages [ 7828.056362] Lustre: server umount lustre-MDT0000 complete [ 7830.496639] LustreError: 248156:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7830.518273] LustreError: 248156:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 7832.456767] LustreError: 248140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786407365 with bad export cookie 13329342567633655146 [ 7832.460573] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7832.477332] LustreError: 248140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7839.087636] Lustre: server umount lustre-MDT0001 complete [ 7850.671204] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 7853.294352] Key type lgssc unregistered [ 7853.542512] LNet: 253263:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7853.557257] LNetError: 253263:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7853.575831] LNet: Removed LNI 192.168.202.104@tcp [ 7854.439986] Key type .llcrypt unregistered [ 7854.442692] Key type ._llcrypt unregistered [ 7901.830090] Key type ._llcrypt registered [ 7901.834588] Key type .llcrypt registered [ 7901.949875] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 7903.651615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7903.717543] alg: No test for adler32 (adler32-zlib) [ 7904.902673] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 7905.162609] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 7906.879188] Key type lgssc registered [ 7908.429291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7917.294473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7917.329757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7918.733168] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7918.777514] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7919.045721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7924.457360] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7933.685174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7933.758252] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7933.780389] Lustre: 255474:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7934.050354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7940.116913] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7949.059435] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7954.807374] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7964.678296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7965.030620] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7965.447435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7971.423935] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7974.333489] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7983.930948] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8037.862736] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8037.864664] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8037.865899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8037.893374] Lustre: Skipped 1 previous similar message [ 8042.977922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8042.986280] Lustre: Skipped 2 previous similar messages [ 8043.786923] Lustre: server umount lustre-OST0000 complete [ 8048.416560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8062.863658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8063.308482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8068.470065] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8068.499718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8068.501476] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8070.805544] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8079.035905] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8119.787306] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8119.796858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8119.802108] Lustre: Skipped 1 previous similar message [ 8119.810412] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8119.817025] Lustre: Skipped 1 previous similar message [ 8124.912993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8124.927659] Lustre: Skipped 1 previous similar message [ 8126.394994] Lustre: server umount lustre-OST0000 complete [ 8133.600903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8133.608684] LustreError: Skipped 1 previous similar message [ 8133.619069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8133.667587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8145.396430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8145.413189] Lustre: Skipped 5 previous similar messages [ 8146.399192] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8146.752675] Lustre: server umount lustre-MDT0000 complete [ 8151.471774] LustreError: 255467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786407684 with bad export cookie 10651675678663765595 [ 8151.477759] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8151.494336] LustreError: 255467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8151.947263] Lustre: server umount lustre-MDT0001 complete [ 8165.141142] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 8167.985458] Key type lgssc unregistered [ 8168.316628] LNet: 260901:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8168.325124] LNetError: 260901:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8168.353372] LNet: Removed LNI 192.168.202.104@tcp [ 8169.368444] Key type .llcrypt unregistered [ 8169.382699] Key type ._llcrypt unregistered [ 8196.892380] Key type ._llcrypt registered [ 8196.894700] Key type .llcrypt registered [ 8197.036756] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 8198.325704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8198.503525] alg: No test for adler32 (adler32-zlib) [ 8199.765653] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 8200.167288] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 8201.938486] Key type lgssc registered [ 8203.120517] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8212.058400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8212.111110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8213.762477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8219.561443] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8223.205044] LustreError: 262322:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8223.227037] LustreError: 262322:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8228.321797] LustreError: 262323:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8228.948316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8229.313775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8233.610931] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8240.975311] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8246.601489] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8253.888161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8254.360152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8258.501668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8260.761587] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8268.650963] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8294.395555] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8294.400810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8294.413322] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8294.426507] Lustre: Skipped 1 previous similar message [ 8294.453069] Lustre: Skipped 1 previous similar message [ 8297.767250] Lustre: server umount lustre-OST0000 complete [ 8304.608941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8304.620473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8304.627184] Lustre: Skipped 1 previous similar message [ 8309.730383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8309.732502] Lustre: Skipped 1 previous similar message [ 8314.848101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8314.860742] Lustre: Skipped 1 previous similar message [ 8316.383165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8316.559237] Lustre: server umount lustre-MDT0000 complete [ 8319.970261] LustreError: 263050:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8319.981356] LustreError: 263050:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8320.192714] LustreError: 262301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786407853 with bad export cookie 2700014761082856885 [ 8320.205632] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8320.503269] Lustre: server umount lustre-MDT0001 complete [ 8331.203996] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 8333.753515] Key type lgssc unregistered [ 8334.109857] LNet: 266629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8334.131232] LNetError: 266629:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8334.158729] LNet: Removed LNI 192.168.202.104@tcp [ 8335.174056] Key type .llcrypt unregistered [ 8335.175763] Key type ._llcrypt unregistered [ 8353.050976] Key type ._llcrypt registered [ 8353.054343] Key type .llcrypt registered [ 8353.154176] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 8354.242147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8354.352459] alg: No test for adler32 (adler32-zlib) [ 8355.489641] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 8355.735852] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 8357.423134] Key type lgssc registered [ 8358.810426] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8366.957131] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8366.979169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8368.688319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8375.010070] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8378.849466] LustreError: 267851:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8378.888670] LustreError: 267851:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8383.992766] LustreError: 267852:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8384.526919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8385.121106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8391.252349] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8400.266609] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8405.723138] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8412.967481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8413.454606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8414.519316] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8419.705595] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8427.535584] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8445.412753] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8445.425388] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8445.441648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8446.432175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8446.441037] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8446.442504] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8448.899274] Lustre: server umount lustre-OST0000 complete [ 8456.686863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8456.691345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8456.719115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8456.721828] Lustre: Skipped 1 previous similar message [ 8459.070535] Lustre: server umount lustre-MDT0000 complete [ 8461.799265] LustreError: 270044:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8461.818553] LustreError: 270044:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8462.573821] LustreError: 267831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786407995 with bad export cookie 11637862841248393396 [ 8462.580332] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8462.584269] LustreError: 267831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8462.903899] Lustre: server umount lustre-MDT0001 complete [ 8473.783424] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 8476.838505] Key type lgssc unregistered [ 8477.138636] LNet: 271531:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8477.146857] LNetError: 271531:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8477.166647] LNet: Removed LNI 192.168.202.104@tcp [ 8478.190588] Key type .llcrypt unregistered [ 8478.192289] Key type ._llcrypt unregistered [ 8502.551638] Key type ._llcrypt registered [ 8502.553870] Key type .llcrypt registered [ 8502.687862] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 8503.982285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8503.998961] alg: No test for adler32 (adler32-zlib) [ 8505.000525] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 8505.217353] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 8506.911285] Key type lgssc registered [ 8507.989908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8517.117536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8517.136358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8518.686876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8523.440110] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8528.879513] LustreError: 272949:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8528.907977] LustreError: 272949:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8532.900510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8533.291575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8538.676288] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8547.059977] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8553.543825] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8560.183017] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8560.521106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8563.608437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8567.979849] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8576.384616] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8608.207615] Lustre: server umount lustre-OST0000 complete [ 8614.885925] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8614.894480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8614.903947] Lustre: Skipped 1 previous similar message [ 8614.920525] Lustre: Skipped 1 previous similar message [ 8620.001133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8623.280084] Lustre: server umount lustre-MDT0000 complete [ 8625.122400] Lustre: 272262:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786408142/real 1786408142] req@00000000182482c3 x1873184594888448/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786408158 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8625.122655] LustreError: 274951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8625.149605] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8625.154726] LustreError: 274951:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8626.144565] Lustre: 272260:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786408142/real 1786408142] req@00000000386abc7b x1873184594888192/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786408158 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8627.877921] LustreError: 273685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786408160 with bad export cookie 6068537279046393369 [ 8627.880097] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8627.901934] LustreError: 273685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8628.445631] Lustre: server umount lustre-MDT0001 complete [ 8641.599109] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 8644.914721] Key type lgssc unregistered [ 8645.279211] LNet: 277259:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8645.289660] LNetError: 277259:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8645.310064] LNet: Removed LNI 192.168.202.104@tcp [ 8646.316277] Key type .llcrypt unregistered [ 8646.320399] Key type ._llcrypt unregistered [ 8664.881595] Key type ._llcrypt registered [ 8664.895459] Key type .llcrypt registered [ 8665.032359] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 8666.295643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8666.378196] alg: No test for adler32 (adler32-zlib) [ 8667.658577] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 8668.077924] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 8669.839493] Key type lgssc registered [ 8671.468534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8680.282791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8680.318484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8682.023815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8687.065381] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8692.198212] LustreError: 278497:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8692.222435] LustreError: 278497:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8695.460756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8695.744528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8701.250923] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8709.845787] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8715.378854] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8722.339776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8722.927475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8727.032430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8729.171700] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8737.874640] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8755.680527] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8755.694918] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8755.701383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8755.725905] Lustre: Skipped 1 previous similar message [ 8755.751377] Lustre: Skipped 1 previous similar message [ 8760.802496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8761.029429] Lustre: server umount lustre-OST0000 complete [ 8765.922545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8765.925656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8765.951645] Lustre: Skipped 1 previous similar message [ 8765.976259] Lustre: Skipped 2 previous similar messages [ 8771.053216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8771.057520] Lustre: Skipped 1 previous similar message [ 8771.675572] Lustre: server umount lustre-MDT0000 complete [ 8775.979957] LustreError: 278464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786408309 with bad export cookie 5613881071871002169 [ 8775.984695] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8775.989015] LustreError: 278464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8776.677390] Lustre: server umount lustre-MDT0001 complete [ 8790.350637] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 8794.077459] Key type lgssc unregistered [ 8794.407837] LNet: 282201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8794.413584] LNetError: 282201:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8794.437665] LNet: Removed LNI 192.168.202.104@tcp [ 8795.401393] Key type .llcrypt unregistered [ 8795.404504] Key type ._llcrypt unregistered [ 8807.870425] Key type ._llcrypt registered [ 8807.872089] Key type .llcrypt registered [ 8808.046385] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8827.372551] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 8845.501684] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 8846.783419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8846.853231] alg: No test for adler32 (adler32-zlib) [ 8848.003880] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 8848.276712] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 8850.039225] Key type lgssc registered [ 8851.200258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8887.691517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8887.717329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8889.054253] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8889.075302] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8889.167393] Lustre: lustre-MDT0000: new disk, initializing [ 8889.313490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8889.332674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8893.976150] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8905.425806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8905.713321] Lustre: 285752:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8905.836466] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8905.841710] Lustre: Skipped 1 previous similar message [ 8905.952408] Lustre: lustre-MDT0001: new disk, initializing [ 8906.089342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8906.151864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8906.166761] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8912.412871] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8923.768907] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8931.882885] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8940.231629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8940.462307] Lustre: lustre-OST0000: new disk, initializing [ 8940.466801] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8940.474721] Lustre: 287719:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8940.546604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8947.201370] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8949.835928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8949.869411] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8949.981405] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8957.557952] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8962.947692] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8963.136110] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8968.277832] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8968.467570] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8970.721238] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8970.728898] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8970.737383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8970.751079] Lustre: Skipped 1 previous similar message [ 8975.842086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8976.508895] Lustre: server umount lustre-OST0000 complete [ 8980.960633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8980.973278] Lustre: Skipped 1 previous similar message [ 8980.979166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8980.982595] Lustre: Skipped 1 previous similar message [ 8986.082201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8986.092926] Lustre: Skipped 1 previous similar message [ 8991.201763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8991.214309] Lustre: Skipped 1 previous similar message [ 8995.296525] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8995.599755] Lustre: server umount lustre-MDT0000 complete [ 8999.723920] LustreError: 287729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786408532 with bad export cookie 9929081320710951881 [ 8999.731749] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8999.737884] LustreError: 287729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9000.219793] Lustre: server umount lustre-MDT0001 complete [ 9009.797484] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 20:35:41 (1786408541) [ 9016.896701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9017.476164] LustreError: 290135:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9017.496602] LustreError: 290135:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9017.543220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9022.736955] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9022.952612] LustreError: 290136:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9027.044065] LustreError: 290135:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9032.161280] LustreError: 290136:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9032.472571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9032.875157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9038.253908] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9045.026920] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9050.670274] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9061.141941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9061.857038] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9069.702813] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9077.985881] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9092.591559] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9092.600268] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9092.606542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9092.620599] Lustre: Skipped 2 previous similar messages [ 9092.632074] Lustre: Skipped 1 previous similar message [ 9094.782966] Lustre: server umount lustre-OST0000 complete [ 9099.233412] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9099.247175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9104.878527] Lustre: server umount lustre-MDT0000 complete [ 9107.936847] LustreError: 291118:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9107.958772] LustreError: 291118:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9109.160743] LustreError: 290117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786408642 with bad export cookie 9929081320710952763 [ 9109.174528] LustreError: 290117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9109.174617] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9109.594550] Lustre: server umount lustre-MDT0001 complete [ 9121.823582] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 9125.098892] Key type lgssc unregistered [ 9125.449337] LNet: 293673:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9125.467931] LNetError: 293673:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9126.502965] LNet: Removed LNI 192.168.202.104@tcp [ 9127.588817] Key type .llcrypt unregistered [ 9127.594351] Key type ._llcrypt unregistered [ 9156.342610] Key type ._llcrypt registered [ 9156.349543] Key type .llcrypt registered [ 9156.474787] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 9173.252115] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 9174.652403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9174.820821] alg: No test for adler32 (adler32-zlib) [ 9176.137743] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 9176.427691] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 9178.111217] Key type lgssc registered [ 9179.537406] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9226.647923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9226.678662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9227.956735] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9228.017076] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9228.126264] Lustre: lustre-MDT0000: new disk, initializing [ 9228.213975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9228.229342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9233.504372] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9244.521185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9244.717206] Lustre: 297507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9244.798625] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9244.802433] Lustre: Skipped 1 previous similar message [ 9244.876987] Lustre: lustre-MDT0001: new disk, initializing [ 9244.980967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9245.019489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9245.047433] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9250.267497] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9260.204930] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9265.406727] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9272.228912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9272.535589] Lustre: lustre-OST2710: new disk, initializing [ 9272.538751] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9272.543532] Lustre: 299476:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9272.644694] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9279.926578] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9282.159744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9282.186181] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9282.277641] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9290.140342] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 9297.222426] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9297.465881] Lustre: lustre-OST03e8: new disk, initializing [ 9297.470059] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9297.480306] Lustre: 300541:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9297.619894] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9304.429713] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9306.278586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9306.302110] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9306.453845] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9312.391534] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 9319.704251] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9319.973904] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9324.893971] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9325.107921] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9329.846183] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9330.063557] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9335.836437] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9336.323964] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9342.441518] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9342.444953] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9342.455932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9342.470116] Lustre: Skipped 1 previous similar message [ 9342.480310] Lustre: Skipped 3 previous similar messages [ 9347.554784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9347.578214] Lustre: Skipped 3 previous similar messages [ 9348.943674] Lustre: server umount lustre-MDT0000 complete [ 9353.104565] LustreError: 297500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786408886 with bad export cookie 18199986702279085334 [ 9353.112279] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9353.422737] Lustre: server umount lustre-MDT0001 complete [ 9366.142574] Lustre: server umount lustre-OST2710 complete [ 9380.790709] Lustre: server umount lustre-OST03e8 complete [ 9399.527904] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 9408.985446] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 9445.079221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9445.434026] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9445.478507] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9445.597383] Lustre: lustre-MDT0000: new disk, initializing [ 9445.705518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9445.720229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9450.923495] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9462.038291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9462.266739] Lustre: 305791:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9462.342408] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9462.347451] Lustre: Skipped 1 previous similar message [ 9462.437308] Lustre: lustre-MDT0001: new disk, initializing [ 9462.557785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9462.620610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9462.631748] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9468.175823] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9477.610443] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9483.348080] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9489.814597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9490.257301] Lustre: lustre-OST0000: new disk, initializing [ 9490.270762] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9490.285313] Lustre: 307762:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9490.443179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9496.691201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9496.702690] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9496.765193] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9496.847405] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9505.598553] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9510.398965] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9510.673911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9515.088092] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9515.302106] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9517.541421] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9517.544419] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9517.551032] LustreError: Skipped 1 previous similar message [ 9517.552405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9517.577161] Lustre: Skipped 3 previous similar messages [ 9522.659101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9522.668220] Lustre: Skipped 2 previous similar messages [ 9527.776761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9527.781112] Lustre: Skipped 1 previous similar message [ 9531.359137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9531.633144] Lustre: server umount lustre-OST0000 complete [ 9538.016674] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9538.027781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9538.042796] Lustre: Skipped 1 previous similar message [ 9538.067846] Lustre: Skipped 1 previous similar message [ 9542.763303] Lustre: server umount lustre-MDT0000 complete [ 9543.144145] LustreError: 305804:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9543.174238] LustreError: 305804:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 9546.675917] LustreError: 305783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786409079 with bad export cookie 18199986702279087028 [ 9546.694255] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9547.100837] Lustre: server umount lustre-MDT0001 complete [ 9556.273493] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 20:44:47 (1786409087) [ 9557.605912] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9559.614273] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 20:44:50 (1786409090) [ 9563.659163] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 9598.427295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9598.603798] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9598.631271] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9598.800740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9603.832934] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9613.730883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9613.923347] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9613.944961] Lustre: 311618:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9619.935726] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9642.177845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9642.492959] Lustre: Denying initial registration attempt from nid 192.168.202.104@tcp, specified as failover [ 9642.497425] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9642.506883] LustreError: 313072:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9642.512902] LustreError: 313072:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9642.517944] LustreError: 313072:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9642.624852] Lustre: server umount lustre-OST0000 complete [ 9642.626936] LustreError: 313072:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9652.202377] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9652.208200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9652.218412] Lustre: Skipped 1 previous similar message [ 9652.230150] Lustre: Skipped 1 previous similar message [ 9656.518653] Lustre: server umount lustre-MDT0000 complete [ 9657.313417] LustreError: 312577:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9657.331568] LustreError: 312577:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9660.257518] LustreError: 311611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786409193 with bad export cookie 18199986702279087910 [ 9660.264510] LustreError: 311611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9660.270567] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9660.756735] Lustre: server umount lustre-MDT0001 complete [ 9672.387714] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 9675.129563] Key type lgssc unregistered [ 9675.524729] LNet: 314195:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9675.531101] LNetError: 314195:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9675.560295] LNet: Removed LNI 192.168.202.104@tcp [ 9676.643066] Key type .llcrypt unregistered [ 9676.646746] Key type ._llcrypt unregistered [ 9688.418695] Key type ._llcrypt registered [ 9688.422931] Key type .llcrypt registered [ 9688.580550] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 9689.787340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9689.904814] alg: No test for adler32 (adler32-zlib) [ 9691.017155] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 9691.242988] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 9692.991233] Key type lgssc registered [ 9694.261464] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9741.961097] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 9750.692976] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9750.710597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9751.926520] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9751.982521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9752.149732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9756.426725] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9764.314175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9764.437676] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9764.463697] Lustre: 316961:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9764.791202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9769.415607] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9777.417359] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9784.100496] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9804.394665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9804.724619] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9804.945298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9811.412698] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9819.313975] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9824.738308] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9824.751647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9824.769080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9827.622400] Lustre: server umount lustre-OST0000 complete [ 9835.501414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9835.509883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9835.517495] Lustre: Skipped 2 previous similar messages [ 9835.530347] Lustre: Skipped 3 previous similar messages [ 9838.010069] Lustre: server umount lustre-MDT0000 complete [ 9841.400734] LustreError: 318828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786409374 with bad export cookie 458028942637443735 [ 9841.423371] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9841.424874] LustreError: 318828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9841.787987] Lustre: server umount lustre-MDT0001 complete [ 9852.110349] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 9855.001594] Key type lgssc unregistered [ 9855.265899] LNet: 320534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9855.273070] LNetError: 320534:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9856.295730] LNet: Removed LNI 192.168.202.104@tcp [ 9857.269194] Key type .llcrypt unregistered [ 9857.272161] Key type ._llcrypt unregistered [ 9884.428431] Key type ._llcrypt registered [ 9884.432519] Key type .llcrypt registered [ 9884.566210] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 9885.731074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9885.754870] alg: No test for adler32 (adler32-zlib) [ 9886.804681] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 9887.008052] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 9888.672575] Key type lgssc registered [ 9889.803772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9897.224370] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9897.263485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9898.741143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9903.574860] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9908.197126] LustreError: 321950:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9908.212621] LustreError: 321950:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9913.325349] LustreError: 321949:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9913.675313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9914.113454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9919.018945] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9926.616599] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9933.553272] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9943.400554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9943.909975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9951.120794] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9959.191096] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9969.642282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9969.655756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9969.656968] Lustre: Skipped 2 previous similar messages [ 9969.669297] Lustre: Skipped 2 previous similar messages [ 9971.088507] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.4@tcp (stopping) [ 9973.681773] Lustre: server umount lustre-MDT0000 complete [ 9974.756497] LustreError: 321949:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9974.789500] LustreError: 321949:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9977.971727] LustreError: 321930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786409511 with bad export cookie 2635158438412564709 [ 9977.981980] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9979.892851] LustreError: 321945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9979.896298] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9979.910132] LustreError: 321945:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9979.937089] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9982.880867] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9982.888223] Lustre: Skipped 1 previous similar message [ 9984.490310] Lustre: server umount lustre-MDT0001 complete [ 9990.942916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9999.281876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10003.424263] LustreError: 321259:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000462561c x1873186044018560/t0(0) o250->MGC192.168.202.104@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10003.901940] LustreError: 323781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10003.929053] LustreError: 323781:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10004.010082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10004.066592] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10009.072252] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [10013.162701] LustreError: 325540:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10013.178352] LustreError: 325540:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10014.197841] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10017.686176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10018.101952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10022.334540] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [10023.416921] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10029.396411] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10031.206778] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10037.356569] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10039.183827] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10048.992764] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10049.000348] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10049.005516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10049.043275] Lustre: Skipped 1 previous similar message [10051.342980] Lustre: server umount lustre-OST0000 complete [10059.238890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10059.243114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10059.250906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10059.250915] Lustre: Skipped 1 previous similar message [10059.276238] Lustre: Skipped 1 previous similar message [10063.306973] Lustre: server umount lustre-MDT0000 complete [10064.357620] LustreError: 325544:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10064.367771] LustreError: 325544:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10067.849880] LustreError: 321930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786409600 with bad export cookie 2635158438412571856 [10067.852630] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10067.854631] LustreError: 321930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10068.100341] Lustre: server umount lustre-MDT0001 complete [10079.770744] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [10082.089499] Key type lgssc unregistered [10082.283829] LNet: 328217:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10082.289503] LNetError: 328217:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10082.308808] LNet: Removed LNI 192.168.202.104@tcp [10082.780254] Key type .llcrypt unregistered [10082.781724] Key type ._llcrypt unregistered [10113.867207] Key type ._llcrypt registered [10113.872449] Key type .llcrypt registered [10114.207817] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [10118.782725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10119.009460] alg: No test for adler32 (adler32-zlib) [10120.658770] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [10121.140226] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [10122.871195] Key type lgssc registered [10125.983546] Lustre: Echo OBD driver; http://www.lustre.org/ [10141.336959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10141.359818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10143.383689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10148.018391] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [10153.953400] LustreError: 329642:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10153.964547] LustreError: 329642:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10158.249858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10158.630617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10163.327740] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [10170.289526] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10177.481972] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10183.078033] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10183.079378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10183.085100] Lustre: Skipped 1 previous similar message [10183.093275] Lustre: Skipped 1 previous similar message [10187.431523] Lustre: server umount lustre-MDT0000 complete [10188.301689] LustreError: 330881:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10188.319961] LustreError: 330881:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10190.932775] LustreError: 329622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786409723 with bad export cookie 11543934323504960166 [10190.945545] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10191.341240] Lustre: server umount lustre-MDT0001 complete [10206.712258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10207.351331] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10207.388581] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10207.625903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10212.985140] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [10222.315304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10222.398945] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10222.434837] Lustre: 332229:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10222.870278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10228.143297] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [10236.276398] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10241.990420] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10252.411988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10252.642618] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10252.660066] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10252.675446] LustreError: 334113:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10252.681872] LustreError: 334113:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10252.689789] LustreError: 334113:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10252.780506] Lustre: server umount lustre-OST0000 complete [10252.782625] LustreError: 334113:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [10263.592471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10263.811714] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10263.927526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10270.885253] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10271.013556] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [10279.451181] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10289.408744] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10289.599526] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10289.771851] Lustre: lustre-OST0001: new disk, initializing [10289.781527] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10289.793652] Lustre: 335657:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10289.958710] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10295.457628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10295.465397] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10295.555371] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10296.321490] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [10304.937584] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10311.136123] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10311.141356] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10311.153088] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10311.163184] Lustre: Skipped 1 previous similar message [10311.170704] Lustre: Skipped 1 previous similar message [10314.614403] Lustre: server umount lustre-OST0001 complete [10316.285738] LustreError: 334581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10316.311511] LustreError: 334581:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10321.379291] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10321.380209] LustreError: 334580:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10321.417150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10321.425732] LustreError: 334580:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10325.342427] Lustre: server umount lustre-OST0000 complete [10330.592566] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10330.612841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10330.640177] Lustre: Skipped 1 previous similar message [10330.656904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10330.670834] Lustre: Skipped 1 previous similar message [10335.868795] Lustre: server umount lustre-MDT0000 complete [10336.748706] LustreError: 334863:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10336.766875] LustreError: 334863:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10339.803051] LustreError: 332222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786409872 with bad export cookie 11543934323504960719 [10339.807714] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10339.820832] LustreError: 332222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10340.182792] Lustre: server umount lustre-MDT0001 complete [10352.630451] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [10356.033669] Key type lgssc unregistered [10356.424132] LNet: 337615:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10356.428334] LNetError: 337615:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10356.447574] LNet: Removed LNI 192.168.202.104@tcp [10357.191401] Key type .llcrypt unregistered [10357.193219] Key type ._llcrypt unregistered [10428.193340] Key type ._llcrypt registered [10428.195669] Key type .llcrypt registered [10428.302589] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [10444.568433] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [10445.855233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10445.885171] alg: No test for adler32 (adler32-zlib) [10446.984609] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [10447.159249] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [10448.783995] Key type lgssc registered [10449.780704] Lustre: Echo OBD driver; http://www.lustre.org/ [10488.771739] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10488.832090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10490.264657] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10490.330348] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10490.444499] Lustre: lustre-MDT0000: new disk, initializing [10490.567177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10490.607394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10495.695905] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [10506.379110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10506.498326] Lustre: 342553:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10506.552336] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10506.556145] Lustre: Skipped 1 previous similar message [10506.697966] Lustre: lustre-MDT0001: new disk, initializing [10506.846326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10506.913174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10506.933755] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10513.215533] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [10522.804504] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10528.586912] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10535.672709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10535.888373] Lustre: lustre-OST0000: new disk, initializing [10535.891705] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10535.896969] Lustre: 344533:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10535.984272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10543.301732] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [10546.270744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10546.280574] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10546.481271] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10553.411844] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10559.189562] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10559.446315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10564.606829] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10564.847979] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10567.141321] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10567.144387] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10567.151252] LustreError: Skipped 1 previous similar message [10567.154333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10567.185268] Lustre: Skipped 1 previous similar message [10572.265137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10572.274060] Lustre: Skipped 2 previous similar messages [10573.557781] Lustre: server umount lustre-OST0000 complete [10578.401622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10578.416380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10578.435946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10582.498499] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10582.500671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10582.526806] Lustre: Skipped 1 previous similar message [10584.489721] Lustre: server umount lustre-MDT0000 complete [10588.466865] LustreError: 344532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786410121 with bad export cookie 16069074660178951208 [10588.473720] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10588.482065] LustreError: 344532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10589.036364] Lustre: server umount lustre-MDT0001 complete [10597.389945] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 21:02:08 (1786410128) [10604.594101] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 21:02:15 (1786410135) [10612.608914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10613.156598] LustreError: 347289:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10613.174208] LustreError: 347289:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10613.231073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10618.337891] LustreError: 347290:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10618.608825] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [10623.464631] LustreError: 347289:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10628.577110] LustreError: 347290:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10629.032886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10629.727336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10635.239720] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [10644.514792] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10651.395444] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10658.405692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10658.801357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10666.364728] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [10674.707566] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10692.634325] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10695.000946] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10697.460523] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10699.468857] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10702.043472] Lustre: Failing over lustre-MDT0000 [10702.685088] Lustre: server umount lustre-MDT0000 complete [10703.848036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10703.855159] LustreError: 349378:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10703.865850] Lustre: Skipped 1 previous similar message [10703.887259] LustreError: 349378:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10714.081874] LustreError: 347286:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10714.103783] LustreError: 347286:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [10719.211952] Lustre: 340779:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786410236/real 1786410236] req@000000003626e0f9 x1873186631262336/t0(0) o400->MGC192.168.202.104@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786410252 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10719.256600] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10724.882669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10729.449107] LustreError: 340776:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004e707bd3 x1873186631267200/t0(0) o250->MGC192.168.202.104@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10730.115646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10730.239561] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10732.421369] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10735.090045] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10735.129641] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10735.221890] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10736.654804] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [10750.670337] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10752.949829] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10756.819690] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10758.449628] Lustre: Failing over lustre-MDT0000 [10758.786972] Lustre: server umount lustre-MDT0000 complete [10760.693737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10760.704812] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10760.706912] LustreError: 351144:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10760.723615] Lustre: Skipped 2 previous similar messages [10760.740264] LustreError: 351144:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [10776.035127] Lustre: 340778:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786410293/real 1786410293] req@00000000b81234a5 x1873186631285888/t0(0) o400->MGC192.168.202.104@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786410309 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10776.077484] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10779.362646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10787.645427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10787.736200] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10788.725748] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10792.488815] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [10792.958930] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10792.971298] Lustre: Skipped 2 previous similar messages [10793.026587] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10793.079415] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10801.209103] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10802.931959] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10813.410423] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10813.412636] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10813.413565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10813.436647] Lustre: Skipped 2 previous similar messages [10814.938344] Lustre: server umount lustre-MDT0000 complete [10818.533305] LustreError: 347290:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10818.558351] LustreError: 347290:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 25 previous similar messages [10819.211348] LustreError: 347270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786410352 with bad export cookie 16069074660178953924 [10819.223792] LustreError: 347270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10819.225200] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10819.562571] Lustre: server umount lustre-MDT0001 complete [10833.259358] Lustre: server umount lustre-OST0000 complete [10852.811478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10853.703734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10859.210264] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [10868.737067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10873.007342] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [10881.228162] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10887.128194] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10895.728394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10898.149461] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10902.936830] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [10911.311173] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10924.013172] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10924.019229] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10924.053531] Lustre: Skipped 1 previous similar message [10924.066862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10924.070134] Lustre: Skipped 2 previous similar messages [10926.987704] Lustre: server umount lustre-OST0000 complete [10937.892560] Lustre: server umount lustre-MDT0000 complete [10940.405079] LustreError: 354382:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10940.431544] LustreError: 354382:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10942.471473] LustreError: 353635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786410475 with bad export cookie 16069074660178954673 [10942.481680] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10942.496179] LustreError: 353635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10943.170811] Lustre: server umount lustre-MDT0001 complete [10958.099930] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [10961.946279] Key type lgssc unregistered [10962.285339] LNet: 357189:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10962.290200] LNetError: 357189:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10962.320320] LNet: Removed LNI 192.168.202.104@tcp [10963.405424] Key type .llcrypt unregistered [10963.409996] Key type ._llcrypt unregistered [10991.923491] Key type ._llcrypt registered [10991.932283] Key type .llcrypt registered [10992.069600] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [10993.453858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10993.491918] alg: No test for adler32 (adler32-zlib) [10994.567923] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [10994.813263] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [10996.463203] Key type lgssc registered [10997.808759] Lustre: Echo OBD driver; http://www.lustre.org/ [11007.026697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11007.061100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11008.749183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11013.961852] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [11018.210631] LustreError: 358707:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11018.241642] LustreError: 358707:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11022.767285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11023.252479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11028.056356] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [11036.390569] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11041.880581] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11049.219725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11049.820734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11053.944308] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11056.625456] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [11065.089737] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11083.753652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11083.756701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11083.767591] Lustre: Skipped 2 previous similar messages [11083.778656] Lustre: Skipped 1 previous similar message [11086.997257] Lustre: server umount lustre-MDT0000 complete [11088.865066] LustreError: 359436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11088.875302] LustreError: 359436:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11091.118315] LustreError: 358687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786410624 with bad export cookie 596481372413969174 [11091.134390] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11091.139249] LustreError: 358687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11091.569743] Lustre: server umount lustre-MDT0001 complete [11106.598979] Lustre: server umount lustre-OST0000 complete [11120.877283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11121.672971] LustreError: 362440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11121.773434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11126.887168] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [11127.281614] LustreError: 362441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11131.429214] LustreError: 362440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11138.074476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11138.764482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11144.951847] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [11152.818846] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11157.905968] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11165.006029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11165.267309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11166.336987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11172.000914] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [11178.529411] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11194.339510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11194.354965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11194.361356] Lustre: Skipped 3 previous similar messages [11196.422904] Lustre: server umount lustre-MDT0000 complete [11199.456086] LustreError: 362441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11199.482363] LustreError: 362441:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11200.041479] LustreError: 362420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786410733 with bad export cookie 596481372413970287 [11200.049965] LustreError: 362420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11200.057866] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11200.288872] Lustre: server umount lustre-MDT0001 complete [11214.548815] Lustre: server umount lustre-OST0000 complete [11233.470498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11234.182855] LustreError: 366352:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11234.329500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11238.786300] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [11248.354718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11254.652163] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [11262.962481] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11269.336628] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11276.551959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11276.925511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11276.937331] Lustre: Skipped 1 previous similar message [11279.983719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11283.507343] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [11291.190613] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11298.800524] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11298.801284] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11298.816052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11298.829186] Lustre: Skipped 3 previous similar messages [11302.474741] Lustre: server umount lustre-OST0000 complete [11309.031314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11309.054065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11309.060334] Lustre: Skipped 1 previous similar message [11312.978539] Lustre: server umount lustre-MDT0000 complete [11314.145597] LustreError: 367080:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11314.165611] LustreError: 367080:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11317.022070] LustreError: 366332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786410850 with bad export cookie 596481372413971309 [11317.029463] LustreError: 366332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11317.041169] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11317.546481] Lustre: server umount lustre-MDT0001 complete [11329.607802] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [11333.018283] Key type lgssc unregistered [11333.236215] LNet: 369885:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11333.244884] LNetError: 369885:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11333.275853] LNet: Removed LNI 192.168.202.104@tcp [11334.522840] Key type .llcrypt unregistered [11334.525294] Key type ._llcrypt unregistered [11362.088177] Key type ._llcrypt registered [11362.089769] Key type .llcrypt registered [11362.203503] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [11363.446729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11363.585456] alg: No test for adler32 (adler32-zlib) [11364.704279] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [11365.139864] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [11366.799338] Key type lgssc registered [11367.855284] Lustre: Echo OBD driver; http://www.lustre.org/ [11375.275720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11375.313416] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11375.318887] LustreError: 371329:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11375.331191] LustreError: 371329:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11375.336779] LustreError: 371329:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11375.346162] LustreError: 371329:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11375.356731] LustreError: 371329:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [11383.411743] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11383.416790] LustreError: 371727:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11383.469686] LustreError: 371727:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11383.472682] LustreError: 371727:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11383.490676] LustreError: 371727:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11383.502142] LustreError: 371727:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [11404.366245] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [11407.372425] Key type lgssc unregistered [11407.710704] LNet: 372730:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11407.725979] LNetError: 372730:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11407.765952] LNet: Removed LNI 192.168.202.104@tcp [11408.659656] Key type .llcrypt unregistered [11408.662551] Key type ._llcrypt unregistered [11426.652113] Key type ._llcrypt registered [11426.653653] Key type .llcrypt registered [11426.763894] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [11444.669862] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [11445.677977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11445.901630] alg: No test for adler32 (adler32-zlib) [11446.939437] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [11447.314840] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [11449.063225] Key type lgssc registered [11450.504908] Lustre: Echo OBD driver; http://www.lustre.org/ [11484.695155] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11484.756275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11486.113418] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11486.139887] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11486.216530] Lustre: lustre-MDT0000: new disk, initializing [11486.285972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11486.312951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11491.871471] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [11503.614621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11503.754753] Lustre: 375889:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11503.797811] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11503.801108] Lustre: Skipped 1 previous similar message [11503.869213] Lustre: lustre-MDT0001: new disk, initializing [11504.028251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11504.073333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11504.089943] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11509.091403] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [11519.030941] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11525.234798] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11533.159845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11533.499385] Lustre: lustre-OST0000: new disk, initializing [11533.504639] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11533.520312] Lustre: 377855:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11533.625856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11539.536162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11539.557627] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11539.765308] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11540.929103] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [11551.633483] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11557.288188] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11557.544485] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11562.276916] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11562.615455] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11565.541555] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11565.545489] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11565.559682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11565.562398] LustreError: Skipped 1 previous similar message [11565.617258] Lustre: Skipped 1 previous similar message [11570.821635] Lustre: server umount lustre-OST0000 complete [11575.777045] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11575.777721] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11575.779323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11575.779329] Lustre: Skipped 1 previous similar message [11575.802491] Lustre: Skipped 1 previous similar message [11580.897084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11580.902058] Lustre: Skipped 2 previous similar messages [11586.017031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11586.024306] Lustre: Skipped 1 previous similar message [11589.087152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11589.421514] Lustre: server umount lustre-MDT0000 complete [11591.145858] LustreError: 375900:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11591.168166] LustreError: 375900:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [11593.217809] LustreError: 375879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786411126 with bad export cookie 17506211836547228835 [11593.225558] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11593.229161] LustreError: 375879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11593.916150] Lustre: server umount lustre-MDT0001 complete [11603.260511] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11604.932521] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 21:18:56 (1786411136) [11612.308521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11612.822206] LustreError: 380319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11612.903186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11617.517167] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [11618.274177] LustreError: 380320:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11622.370838] LustreError: 380319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11627.468238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11627.497294] LustreError: 380320:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11628.215198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11634.164753] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [11642.254361] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11648.219925] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11656.991960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11657.132498] LustreError: 382139:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11657.236166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11661.416214] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [11669.295835] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11676.976637] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11677.154283] Lustre: lustre-OST0001: new disk, initializing [11677.158347] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11677.164568] Lustre: 383156:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11677.255847] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11684.471631] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [11684.476143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11684.484480] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11684.615560] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11694.073702] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [11699.752642] Lustre: lustre-OST0001: Not available for connect from 192.168.202.4@tcp (stopping) [11700.197620] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11700.199564] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11704.237242] Lustre: server umount lustre-OST0001 complete [11705.212352] LustreError: 382996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11723.747083] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11723.766128] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11723.784760] Lustre: Skipped 1 previous similar message [11723.803894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11723.808910] Lustre: Skipped 2 previous similar messages [11728.522605] Lustre: server umount lustre-OST0000 complete [11735.520874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11735.537540] LustreError: Skipped 1 previous similar message [11735.540364] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11735.553980] Lustre: Skipped 1 previous similar message [11738.620112] Lustre: server umount lustre-MDT0000 complete [11741.158361] LustreError: 380319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11741.172208] LustreError: 380319:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [11742.447081] LustreError: 380300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786411275 with bad export cookie 17506211836547229717 [11742.463564] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11749.376968] Lustre: server umount lustre-MDT0001 complete [11761.482567] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [11764.320342] Key type lgssc unregistered [11764.603164] LNet: 385142:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11764.608853] LNetError: 385142:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11764.624799] LNet: Removed LNI 192.168.202.104@tcp [11765.712288] Key type .llcrypt unregistered [11765.717302] Key type ._llcrypt unregistered [11825.393845] Key type ._llcrypt registered [11825.396658] Key type .llcrypt registered [11825.518265] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [11826.525166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11826.548335] alg: No test for adler32 (adler32-zlib) [11828.496770] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [11829.006315] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [11830.928364] Key type lgssc registered [11832.215077] Lustre: Echo OBD driver; http://www.lustre.org/ [11840.893466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11840.918860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11842.180745] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11842.197827] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11842.430130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11847.428867] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [11857.264830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11857.388754] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11857.476758] Lustre: 387957:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11857.830313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11863.814278] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [11871.712455] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11878.331624] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11883.492878] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11883.507498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11883.524667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11886.975503] Lustre: server umount lustre-MDT0000 complete [11891.022439] LustreError: 387952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786411424 with bad export cookie 7623317591066338813 [11891.035702] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11891.594639] Lustre: server umount lustre-MDT0001 complete [11902.601619] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 21:23:53 (1786411433) [11909.701793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11910.270570] LustreError: 390471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11910.284280] LustreError: 390471:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11910.341290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11915.737564] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [11915.747742] LustreError: 390472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11920.876612] LustreError: 390471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11924.973321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11925.134321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11929.811523] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [11938.514253] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11946.240615] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11953.478033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11953.612579] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11953.774022] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11959.913080] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [11968.233236] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11975.945707] Lustre: Permanently deactivating lustre-OST0000 [11975.963280] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11985.996436] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11987.727578] LustreError: 393509:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11987.747881] LustreError: 393509:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [11987.761535] LustreError: 393509:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11991.495158] LustreError: 393558:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [11991.503396] LustreError: 393558:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11999.734399] Lustre: server umount lustre-OST0000 complete [12002.727271] LustreError: 393809:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [12002.731039] LustreError: 393809:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12007.401610] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12007.409413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12007.420372] Lustre: Skipped 1 previous similar message [12007.428394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12007.436186] Lustre: Skipped 1 previous similar message [12013.024801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12013.032183] Lustre: Skipped 2 previous similar messages [12014.200886] Lustre: server umount lustre-MDT0000 complete [12018.148451] LustreError: 390472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12018.170157] LustreError: 390472:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12018.367511] LustreError: 390451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786411551 with bad export cookie 7623317591066339394 [12018.370659] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12018.382236] LustreError: 390451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [12029.027340] Lustre: server umount lustre-MDT0001 complete [12038.533149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12043.123274] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [12046.645020] LustreError: 394967:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12054.407219] Lustre: 395306:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.104@tcp [12056.841342] Lustre: server umount MGS complete [12068.392381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12068.971699] LustreError: 396030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12068.989114] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12068.999295] Lustre: Skipped 1 previous similar message [12069.081340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12074.264956] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [12079.588608] LustreError: 396031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12079.610097] LustreError: 396031:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12083.839864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12084.097656] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12089.239394] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [12096.823902] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12102.960850] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12105.041290] Lustre: Permanently reactivating lustre-OST0000 [12105.055141] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12105.060150] Lustre: Skipped 2 previous similar messages [12135.396227] LustreError: 397860:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12143.540992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12144.084962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12144.095391] Lustre: Skipped 1 previous similar message [12150.717303] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [12158.121874] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12165.126062] Lustre: server umount lustre-OST0000 complete [12170.727486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12170.736938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12170.753213] Lustre: Skipped 2 previous similar messages [12170.771353] Lustre: Skipped 1 previous similar message [12175.846933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12175.856743] Lustre: Skipped 1 previous similar message [12178.967040] Lustre: server umount lustre-MDT0000 complete [12180.960866] LustreError: 396027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12180.984035] LustreError: 396027:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12181.983267] Lustre: 387282:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786411698/real 1786411698] req@00000000c7a32197 x1873188080419456/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786411714 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12182.032894] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12182.708474] LustreError: 396010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786411715 with bad export cookie 7623317591066340444 [12182.714655] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12182.723527] LustreError: 396010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12183.046785] Lustre: server umount lustre-MDT0001 complete [12194.362599] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [12198.752531] Key type lgssc unregistered [12199.167512] LNet: 400493:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12199.175467] LNetError: 400493:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12199.195346] LNet: Removed LNI 192.168.202.104@tcp [12200.160840] Key type .llcrypt unregistered [12200.162382] Key type ._llcrypt unregistered [12221.063872] Key type ._llcrypt registered [12221.067743] Key type .llcrypt registered [12221.170664] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [12237.408061] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [12238.696356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12238.752094] alg: No test for adler32 (adler32-zlib) [12239.791845] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [12240.082047] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [12241.767981] Key type lgssc registered [12242.624386] Lustre: Echo OBD driver; http://www.lustre.org/ [12281.139283] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12281.165245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12282.691097] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12282.762390] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12282.874455] Lustre: lustre-MDT0000: new disk, initializing [12282.994040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12283.007633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12288.339300] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [12299.048975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12299.260734] Lustre: 403650:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12299.327534] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12299.331627] Lustre: Skipped 1 previous similar message [12299.403549] Lustre: lustre-MDT0001: new disk, initializing [12299.527937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12299.575458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12299.587968] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12304.629687] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [12314.267375] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12321.245962] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12328.635328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12328.875582] Lustre: lustre-OST0000: new disk, initializing [12328.882971] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12328.888362] Lustre: 405619:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12329.001962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12335.899875] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [12336.200751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12336.210861] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12336.329948] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12346.964716] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12352.251296] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12352.483293] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12357.925183] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12358.244431] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12362.218674] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12362.227810] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12362.269578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12366.741077] Lustre: server umount lustre-OST0000 complete [12371.423948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12371.441511] LustreError: Skipped 1 previous similar message [12371.447736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12371.463262] Lustre: Skipped 1 previous similar message [12371.470546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12371.472734] Lustre: Skipped 1 previous similar message [12372.455495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12372.472263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12372.477423] Lustre: Skipped 1 previous similar message [12377.572205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12377.828749] Lustre: server umount lustre-MDT0000 complete [12381.607886] LustreError: 405618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786411914 with bad export cookie 5235381663946748348 [12381.611966] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12381.615416] LustreError: 405618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12381.879478] Lustre: server umount lustre-MDT0001 complete [12391.360576] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12393.105290] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 21:32:04 (1786411924) [12401.924464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12402.768615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12407.776976] LustreError: 408078:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12407.802072] LustreError: 408078:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12408.341242] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [12412.897117] LustreError: 408079:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12418.016239] LustreError: 408078:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12418.363517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12418.716799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12425.095602] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [12433.513972] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12439.242230] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12447.109163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12447.725808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12454.752638] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [12462.152884] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12464.009112] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12466.785801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12478.439441] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12478.443020] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12478.464751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12478.464760] Lustre: Skipped 1 previous similar message [12478.526284] Lustre: Skipped 1 previous similar message [12480.794460] Lustre: server umount lustre-OST0000 complete [12488.683164] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12488.689342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12488.698885] Lustre: Skipped 1 previous similar message [12488.705986] Lustre: Skipped 1 previous similar message [12491.710834] Lustre: server umount lustre-MDT0000 complete [12493.797588] LustreError: 409005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12493.830319] LustreError: 409005:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12495.406315] LustreError: 408061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786412028 with bad export cookie 5235381663946749230 [12495.413095] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12496.096946] Lustre: server umount lustre-MDT0001 complete [12510.107931] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [12513.492911] Key type lgssc unregistered [12513.882232] LNet: 411807:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12513.902631] LNetError: 411807:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12513.938224] LNet: Removed LNI 192.168.202.104@tcp [12514.618537] Key type .llcrypt unregistered [12514.623577] Key type ._llcrypt unregistered [12543.114863] Key type ._llcrypt registered [12543.117323] Key type .llcrypt registered [12543.258493] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [12543.638432] Key type .llcrypt unregistered [12543.641606] Key type ._llcrypt unregistered [12566.005478] Key type ._llcrypt registered [12566.008905] Key type .llcrypt registered [12566.188155] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [12567.539312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12567.571769] alg: No test for adler32 (adler32-zlib) [12568.699402] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [12568.963381] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [12570.639165] Key type lgssc registered [12571.762871] Lustre: Echo OBD driver; http://www.lustre.org/ [12580.066248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12580.086754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12581.655580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12586.043508] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [12591.089497] LustreError: 414024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12591.129170] LustreError: 414024:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12595.008325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12595.469033] LustreError: 414023:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12595.486712] LustreError: 414023:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12595.513146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12600.816542] LustreError: 414024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12600.849361] LustreError: 414024:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12601.925658] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [12605.925745] LustreError: 414020:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12605.940582] LustreError: 414020:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12606.977858] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12610.914461] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12612.065957] LustreError: 414020:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12612.075093] LustreError: 414020:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12616.703314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12617.152218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12621.887423] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [12632.557649] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12632.565179] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12632.583857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12632.586571] LustreError: Skipped 1 previous similar message [12632.634484] Lustre: Skipped 1 previous similar message [12637.679318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12637.682987] Lustre: Skipped 1 previous similar message [12642.789803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12642.794429] Lustre: Skipped 1 previous similar message [12646.367230] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12646.850821] Lustre: server umount lustre-OST0000 complete [12653.026986] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12653.053171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12653.055967] Lustre: Skipped 2 previous similar messages [12658.146876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12658.159540] Lustre: Skipped 1 previous similar message [12658.791866] Lustre: server umount lustre-MDT0000 complete [12663.270187] LustreError: 414020:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12663.296391] LustreError: 414020:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12663.790209] LustreError: 414005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786412196 with bad export cookie 1282193342135460660 [12663.795397] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12663.809702] LustreError: 414005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12664.296505] Lustre: server umount lustre-MDT0001 complete [12675.999475] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [12678.637956] Key type lgssc unregistered [12678.830400] LNet: 417174:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12678.835796] LNetError: 417174:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12678.855134] LNet: Removed LNI 192.168.202.104@tcp [12679.758107] Key type .llcrypt unregistered [12679.768705] Key type ._llcrypt unregistered [12710.266356] Key type ._llcrypt registered [12710.271696] Key type .llcrypt registered [12710.444945] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [12712.031382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12712.063222] alg: No test for adler32 (adler32-zlib) [12713.398924] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [12713.682219] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [12715.415481] Key type lgssc registered [12716.676717] Lustre: Echo OBD driver; http://www.lustre.org/ [12725.058900] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12725.117380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12740.831573] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12746.015344] LustreError: 418564:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.104@tcp: failed processing log, type 4: rc = -110 [12772.708807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12779.604558] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [12787.611124] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12794.274240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12794.706483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12794.778914] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12799.180849] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [12799.971969] LustreError: 418590:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12800.005692] LustreError: 418590:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12805.091305] LustreError: 418589:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12807.154708] LustreError: 419601:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12807.174598] LustreError: 419601:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12807.897635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12808.196737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12812.892919] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [12820.304482] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12825.007672] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12833.783218] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12833.803087] LustreError: 418578:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12833.817423] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12834.113621] Lustre: server umount lustre-OST0000 complete [12838.824093] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12838.828846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12838.838146] Lustre: Skipped 1 previous similar message [12838.843709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12841.530670] Lustre: server umount lustre-MDT0000 complete [12844.000808] LustreError: 419606:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12844.016624] LustreError: 419606:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12845.949079] LustreError: 418570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786412378 with bad export cookie 10155772330677265794 [12845.952819] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12845.971073] LustreError: 418570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12846.647918] Lustre: server umount lustre-MDT0001 complete [12860.689100] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [12863.047921] Key type lgssc unregistered [12863.253916] LNet: 422132:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12863.259436] LNetError: 422132:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12863.286639] LNet: Removed LNI 192.168.202.104@tcp [12864.322279] Key type .llcrypt unregistered [12864.329272] Key type ._llcrypt unregistered [12891.040795] Key type ._llcrypt registered [12891.042467] Key type .llcrypt registered [12891.153680] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [12892.143245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12892.313795] alg: No test for adler32 (adler32-zlib) [12893.356303] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [12893.601911] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [12895.287210] Key type lgssc registered [12897.063784] Lustre: Echo OBD driver; http://www.lustre.org/ [12908.442400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12908.469411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12910.325376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12916.600852] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [12920.803208] LustreError: 423559:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12920.831973] LustreError: 423559:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12921.827525] LustreError: 423560:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12926.961157] LustreError: 423559:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12932.064628] LustreError: 423560:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12932.171754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12932.687454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12938.034661] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [12947.529626] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12954.322539] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12961.757844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12962.166828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12969.269718] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [12977.515498] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12981.626351] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12986.855748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12986.885128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12986.899058] Lustre: Skipped 2 previous similar messages [12988.322226] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.4@tcp (stopping) [12991.970163] LustreError: 423554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12991.980906] LustreError: 423554:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12992.032485] Lustre: server umount lustre-MDT0000 complete [12994.456197] LustreError: 423539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.4@tcp arrived at 1786412527 with bad export cookie 18425760503218961763 [12996.234634] LustreError: 423540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786412529 with bad export cookie 18425760503218961098 [12996.240703] LustreError: 423540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12996.241408] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12997.088849] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12997.094332] Lustre: Skipped 2 previous similar messages [12997.097344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13001.123894] LustreError: 423554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13001.126319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13001.141039] LustreError: 423554:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [13002.610549] Lustre: server umount lustre-OST0000 complete [13009.495642] Lustre: server umount lustre-MDT0001 complete [13021.652721] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [13024.545367] Key type lgssc unregistered [13025.068550] LNet: 427249:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13025.079972] LNetError: 427249:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13025.103162] LNet: Removed LNI 192.168.202.104@tcp [13026.311603] Key type .llcrypt unregistered [13026.315242] Key type ._llcrypt unregistered [13053.154411] Key type ._llcrypt registered [13053.157636] Key type .llcrypt registered [13053.234277] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [13054.337106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13054.420892] alg: No test for adler32 (adler32-zlib) [13055.775651] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [13056.049160] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [13057.704553] Key type lgssc registered [13058.847595] Lustre: Echo OBD driver; http://www.lustre.org/ [13067.751150] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13067.782728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13069.391330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13074.128431] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [13079.521099] LustreError: 428666:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13079.552096] LustreError: 428666:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13084.640939] LustreError: 428665:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13086.127478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13086.368165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13093.598212] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [13104.663095] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13112.802799] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13120.233957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13120.565144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13128.216359] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [13138.323903] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13138.995559] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13148.128902] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13148.131455] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13148.162411] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13149.074182] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.4@tcp (stopping) [13149.159424] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13152.505512] Lustre: server umount lustre-MDT0001 complete [13154.169839] LustreError: 430963:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13159.392465] LustreError: 429581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13159.397253] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13159.402360] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13159.403122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13159.403127] Lustre: Skipped 2 previous similar messages [13159.418240] LustreError: 429581:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13164.515211] LustreError: 428666:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13164.515252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13165.029450] Lustre: server umount lustre-OST0000 complete [13170.267657] Lustre: server umount lustre-MDT0000 complete [13186.661801] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [13190.560585] Key type lgssc unregistered [13190.969935] LNet: 432353:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13190.980236] LNetError: 432353:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13191.005994] LNet: Removed LNI 192.168.202.104@tcp [13192.102987] Key type .llcrypt unregistered [13192.105319] Key type ._llcrypt unregistered [13223.393538] Key type ._llcrypt registered [13223.409924] Key type .llcrypt registered [13223.604648] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [13242.209269] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [13243.801803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13243.831092] alg: No test for adler32 (adler32-zlib) [13244.939260] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [13245.246096] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [13246.992391] Key type lgssc registered [13248.356125] Lustre: Echo OBD driver; http://www.lustre.org/ [13288.253266] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [13304.131926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13304.181923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13305.624133] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13305.642776] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13305.712996] Lustre: lustre-MDT0000: new disk, initializing [13305.790738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13305.816547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13310.951606] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [13328.107823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13328.510704] Lustre: lustre-OST0000: new disk, initializing [13328.514870] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13328.518407] Lustre: Skipped 1 previous similar message [13328.522984] Lustre: 437393:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13328.627266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13334.136322] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [13338.234851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13338.260264] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13338.353402] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13345.002973] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13361.135710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13361.292621] Lustre: 436373:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13361.354312] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13361.424758] Lustre: lustre-MDT0001: new disk, initializing [13361.592363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13361.656295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13361.667991] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13366.388831] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [13376.191412] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13376.370766] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13380.999136] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13381.214400] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13387.690873] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.4@tcp (stopping) [13388.256708] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13388.265027] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13388.284986] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13388.295108] Lustre: Skipped 1 previous similar message [13392.784067] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.4@tcp (stopping) [13392.794381] Lustre: Skipped 1 previous similar message [13393.193805] Lustre: server umount lustre-MDT0001 complete [13398.501992] LustreError: 436386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13398.530878] LustreError: 436386:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [13403.619518] LustreError: 436380:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13403.654763] LustreError: 436380:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13408.747242] LustreError: 436381:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13408.779656] LustreError: 436381:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13413.863174] LustreError: 436385:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13413.863343] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13413.878730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13418.979266] LustreError: 439250:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13418.981795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13426.143636] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13426.325897] Lustre: server umount lustre-OST0000 complete [13429.220740] LustreError: 439250:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13429.243197] LustreError: 439250:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13437.202389] Lustre: server umount lustre-MDT0000 complete [13454.066989] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [13457.413640] Key type lgssc unregistered [13457.791712] LNet: 441143:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13457.798631] LNetError: 441143:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13457.818570] LNet: Removed LNI 192.168.202.104@tcp [13458.968338] Key type .llcrypt unregistered [13458.970050] Key type ._llcrypt unregistered [13500.262190] Key type ._llcrypt registered [13500.264334] Key type .llcrypt registered [13500.376357] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [13500.634382] Key type .llcrypt unregistered [13500.642181] Key type ._llcrypt unregistered [13519.441942] Key type ._llcrypt registered [13519.443499] Key type .llcrypt registered [13519.584276] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [13521.208104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13521.227436] alg: No test for adler32 (adler32-zlib) [13522.432526] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [13522.644516] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [13524.407158] Key type lgssc registered [13525.566176] Lustre: Echo OBD driver; http://www.lustre.org/ [13550.559336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13550.593420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13551.815703] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13551.855782] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13551.907593] Lustre: lustre-MDT0000: new disk, initializing [13551.961718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13551.996600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13557.387222] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [13569.610257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13569.772903] Lustre: 444522:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13569.901981] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13569.906702] Lustre: Skipped 1 previous similar message [13570.012824] Lustre: lustre-MDT0001: new disk, initializing [13570.157431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13570.194709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13570.213022] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13575.686876] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [13585.781258] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13592.790259] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13598.777715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13598.978686] Lustre: lustre-OST0000: new disk, initializing [13598.980432] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13598.983124] Lustre: 446490:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13599.047328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13605.087268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13605.119258] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13605.309882] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13606.563259] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [13618.882225] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13626.336713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13626.339891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13626.347925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13630.944097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13630.947935] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13632.287344] Lustre: server umount lustre-MDT0000 complete [13636.462348] LustreError: 444513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786413169 with bad export cookie 12992131131526467441 [13636.464962] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13636.476909] LustreError: 444513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13636.713864] Lustre: server umount lustre-MDT0001 complete [13649.329870] Lustre: server umount lustre-OST0000 complete [13660.390402] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 21:53:11 (1786413191) [13668.162393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13668.717201] LustreError: 448520:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13668.740535] LustreError: 448520:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [13668.805761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13673.521958] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [13673.963079] LustreError: 448521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13678.062418] LustreError: 448520:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13683.175757] LustreError: 448521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13684.061528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13684.859564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13689.830801] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [13698.391373] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13704.893905] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13725.994970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13726.231837] Lustre: Found index 0 for lustre-OST0000, updating log [13726.481763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13732.783527] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [13739.821723] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13740.031390] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13745.120592] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13745.132673] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13745.136406] Lustre: Skipped 1 previous similar message [13745.138875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13745.138882] Lustre: Skipped 2 previous similar messages [13745.173765] LustreError: Skipped 1 previous similar message [13747.146385] Lustre: server umount lustre-OST0000 complete [13766.626146] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13766.635663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13766.642438] Lustre: Skipped 1 previous similar message [13766.645459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13766.649102] Lustre: Skipped 3 previous similar messages [13768.857168] Lustre: server umount lustre-MDT0000 complete [13772.769218] LustreError: 448521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13772.793190] LustreError: 448521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13773.218257] LustreError: 448502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786413306 with bad export cookie 12992131131526468834 [13773.228395] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13773.552410] Lustre: server umount lustre-MDT0001 complete [13781.542947] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 21:55:12 (1786413312) [13786.912434] LNet: 1 local NIs in recovery (showing 1): 192.168.202.104@tcp [14406.863267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14407.581794] LustreError: 454941:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14407.774259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14412.788976] LustreError: 454942:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14415.687557] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [14417.893251] LustreError: 454941:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14424.032792] LustreError: 454942:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14424.044102] LustreError: 454942:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14425.117117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14425.406832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14430.928662] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [14437.687662] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14443.041846] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14462.275553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14462.579213] Lustre: Found index 0 for lustre-OST0000, updating log [14462.984455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14463.776573] LNet: 1 local NIs in recovery (showing 1): 192.168.202.104@tcp [14469.053565] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [14476.217163] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14478.454969] Lustre: server umount lustre-OST0000 complete [14483.051518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14483.068263] Lustre: Skipped 1 previous similar message [14483.072530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14483.091434] Lustre: Skipped 3 previous similar messages [14484.129400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14484.148101] Lustre: Skipped 1 previous similar message [14485.152954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14485.161046] Lustre: Skipped 1 previous similar message [14487.207311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14487.214040] Lustre: Skipped 3 previous similar messages [14488.398813] Lustre: server umount lustre-MDT0000 complete [14489.251102] LustreError: 455672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14489.260054] LustreError: 455672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14491.655202] LustreError: 454923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786414024 with bad export cookie 12992131131526469919 [14491.659561] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14491.666229] LustreError: 454923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14491.986446] Lustre: server umount lustre-MDT0001 complete [14521.903322] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [14523.167168] LNet: 1 local NIs in recovery (showing 1): 192.168.202.104@tcp [14534.331630] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [14575.417033] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 22:08:27 (1786414107) [14595.734594] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [14598.585356] Key type lgssc unregistered [14598.783496] LNet: 461550:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14598.787236] LNetError: 461550:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14722.857545] LNet: Removed LNI 192.168.202.104@tcp [14724.402923] Key type .llcrypt unregistered [14724.405151] Key type ._llcrypt unregistered [14742.473445] Key type ._llcrypt registered [14742.476064] Key type .llcrypt registered [14742.600884] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [14743.907679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14744.010141] alg: No test for adler32 (adler32-zlib) [14745.153698] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [14745.441241] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [14747.223215] Key type lgssc registered [14748.615623] Lustre: Echo OBD driver; http://www.lustre.org/ [14753.150379] LNet: Added LNI 192.168.202.104@tcp100 [8/256/0/180] [14754.014047] LNet: Added LNI 192.168.202.104@tcp101 [8/256/0/180] [14755.150574] LNet: Added LNI 192.168.202.104@tcp102 [8/256/0/180] [14755.951835] LNet: Added LNI 192.168.202.104@tcp103 [8/256/0/180] [14756.921215] LNet: Added LNI 192.168.202.104@tcp104 [8/256/0/180] [14757.765098] LNet: Added LNI 192.168.202.104@tcp105 [8/256/0/180] [14758.685551] LNet: Added LNI 192.168.202.104@tcp106 [8/256/0/180] [14759.536628] LNet: Added LNI 192.168.202.104@tcp107 [8/256/0/180] [14760.544099] LNet: Added LNI 192.168.202.104@tcp108 [8/256/0/180] [14761.548685] LNet: Added LNI 192.168.202.104@tcp109 [8/256/0/180] [14762.616473] LNet: Added LNI 192.168.202.104@tcp110 [8/256/0/180] [14763.517908] LNet: Added LNI 192.168.202.104@tcp111 [8/256/0/180] [14764.740125] LNet: Added LNI 192.168.202.104@tcp112 [8/256/0/180] [14765.736249] LNet: Added LNI 192.168.202.104@tcp113 [8/256/0/180] [14766.627615] LNet: Added LNI 192.168.202.104@tcp114 [8/256/0/180] [14767.561899] LNet: Added LNI 192.168.202.104@tcp115 [8/256/0/180] [14768.759808] LNet: Added LNI 192.168.202.104@tcp116 [8/256/0/180] [14769.989559] LNet: Added LNI 192.168.202.104@tcp117 [8/256/0/180] [14771.263104] LNet: Added LNI 192.168.202.104@tcp118 [8/256/0/180] [14772.360631] LNet: Added LNI 192.168.202.104@tcp119 [8/256/0/180] [14773.260783] LNet: Added LNI 192.168.202.104@tcp120 [8/256/0/180] [14774.161925] LNet: Added LNI 192.168.202.104@tcp121 [8/256/0/180] [14775.017400] LNet: Added LNI 192.168.202.104@tcp122 [8/256/0/180] [14776.257896] LNet: Added LNI 192.168.202.104@tcp123 [8/256/0/180] [14777.307797] LNet: Added LNI 192.168.202.104@tcp124 [8/256/0/180] [14778.805127] LNet: Added LNI 192.168.202.104@tcp125 [8/256/0/180] [14779.894976] LNet: Added LNI 192.168.202.104@tcp126 [8/256/0/180] [14781.100273] LNet: Added LNI 192.168.202.104@tcp127 [8/256/0/180] [14782.153196] LNet: Added LNI 192.168.202.104@tcp128 [8/256/0/180] [14782.964607] LNet: Added LNI 192.168.202.104@tcp129 [8/256/0/180] [14783.837967] LNet: Added LNI 192.168.202.104@tcp130 [8/256/0/180] [14784.843050] LNet: Added LNI 192.168.202.104@tcp131 [8/256/0/180] [14785.911476] LNet: Added LNI 192.168.202.104@tcp132 [8/256/0/180] [14787.037213] LNet: Added LNI 192.168.202.104@tcp133 [8/256/0/180] [14788.080942] LNet: Added LNI 192.168.202.104@tcp134 [8/256/0/180] [14789.032821] LNet: Added LNI 192.168.202.104@tcp135 [8/256/0/180] [14797.538763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14797.582387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14798.761669] Lustre: 464586:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.104@tcp: can't add NID 192.168.202.104@tcp130: rc = -75 [14798.868162] LNetError: 462043:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.104 [14798.891719] LNetError: Protocol error connecting to 192.168.202.104@tcp120 at host 192.168.202.104:988: is it running a compatible version of Lustre? [14798.912662] LNetError: Skipped 8 previous similar messages [14799.090188] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14799.132876] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14799.253678] Lustre: lustre-MDT0000: new disk, initializing [14799.345632] Lustre: 464642:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.104@tcp: can't add NID 192.168.202.104@tcp130: rc = -75 [14799.356880] Lustre: 464642:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14799.372803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14799.399644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14799.712527] LNetError: 464610:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.104 [14799.714489] LNet: 8 local NIs in recovery (showing 8): 192.168.202.104@tcp123, 192.168.202.104@tcp125, 192.168.202.104@tcp124, 192.168.202.104@tcp126, 192.168.202.104@tcp127, 192.168.202.104@tcp129, 192.168.202.104@tcp128, 192.168.202.104@tcp120 [14799.724300] LNetError: 464610:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [14799.729297] LNetError: Protocol error connecting to 192.168.202.104@tcp101 at host 192.168.202.104:988: is it running a compatible version of Lustre? [14799.749827] LNetError: Skipped 6 previous similar messages [14803.842802] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [14814.723638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14814.935569] Lustre: 464603:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14815.020711] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14815.026489] Lustre: Skipped 1 previous similar message [14815.138334] Lustre: lustre-MDT0001: new disk, initializing [14815.219993] Lustre: 465405:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.104@tcp: can't add NID 192.168.202.104@tcp130: rc = -75 [14815.228676] Lustre: 465405:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14815.426558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14815.494126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14815.512084] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14819.967831] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [14823.991766] Lustre: 465829:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.104@tcp: can't add NID 192.168.202.104@tcp130: rc = -75 [14823.998682] Lustre: 465829:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14829.421501] LNetError: 462043:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.104 [14829.445927] LNetError: 462043:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 5 previous similar messages [14829.467562] LNetError: Protocol error connecting to 192.168.202.104@tcp128 at host 192.168.202.104:988: is it running a compatible version of Lustre? [14829.953479] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14834.533511] LNetError: 464615:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.104 [14834.567029] LNetError: 464615:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [14834.590460] LNetError: Protocol error connecting to 192.168.202.104@tcp129 at host 192.168.202.104:988: is it running a compatible version of Lustre? [14834.618678] LNetError: Skipped 2 previous similar messages [14837.212564] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14838.628315] LNetError: 464606:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.104 [14838.647979] LNetError: 464606:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [14838.655933] LNetError: Protocol error connecting to 192.168.202.104@tcp117 at host 192.168.202.104:988: is it running a compatible version of Lustre? [14838.691227] LNetError: Skipped 3 previous similar messages [14845.248150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14845.434352] Lustre: lustre-OST0000: new disk, initializing [14845.437607] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14845.445185] Lustre: 466576:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14845.528784] Lustre: 466581:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.104@tcp: can't add NID 192.168.202.104@tcp130: rc = -75 [14845.536561] Lustre: 466581:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14845.578651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14846.817638] LNetError: 462042:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.104 [14846.827513] LNetError: 462042:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 4 previous similar messages [14846.832663] LNetError: Protocol error connecting to 192.168.202.104@tcp129 at host 192.168.202.104:988: is it running a compatible version of Lustre? [14846.843630] LNetError: Skipped 4 previous similar messages [14852.199453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14852.215107] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14852.373484] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [14852.427212] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14862.924576] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14863.200393] LNetError: 464619:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.104 [14863.209593] LNetError: 464619:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 9 previous similar messages [14863.219217] LNetError: Protocol error connecting to 192.168.202.104@tcp117 at host 192.168.202.104:988: is it running a compatible version of Lustre? [14863.228994] LNetError: Skipped 9 previous similar messages [14873.056941] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14873.059340] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14873.071922] LustreError: Skipped 1 previous similar message [14873.081785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14873.095293] Lustre: Skipped 1 previous similar message [14873.116570] Lustre: Skipped 1 previous similar message [14878.191152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14878.197349] Lustre: Skipped 1 previous similar message [14883.297678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14883.314060] Lustre: Skipped 1 previous similar message [14885.343647] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14885.672665] Lustre: server umount lustre-OST0000 complete [14892.001887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14892.009080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14892.026218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14893.539321] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14895.687784] Lustre: server umount lustre-MDT0000 complete [14898.016384] LNetError: 464606:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.104 [14898.023312] LNetError: 464606:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [14898.027184] LNetError: Protocol error connecting to 192.168.202.104@tcp129 at host 192.168.202.104:988: is it running a compatible version of Lustre? [14898.032714] LNetError: Skipped 10 previous similar messages [14899.203671] LustreError: 464595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786414432 with bad export cookie 15973685699035309939 [14899.203928] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14899.209800] LustreError: 464595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14899.554214] Lustre: server umount lustre-MDT0001 complete [14911.596793] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [14914.583262] Key type lgssc unregistered [14915.011392] LNet: 468490:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14915.020453] LNetError: 468490:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14915.027760] LNet: Removed LNI 192.168.202.104@tcp [14916.064113] LNet: Removed LNI 192.168.202.104@tcp100 [14916.068750] LNetError: 468490:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14916.075718] LNetError: 468490:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14916.090597] LNet: Removed LNI 192.168.202.104@tcp101 [14917.153487] LNet: Removed LNI 192.168.202.104@tcp102 [14917.155354] LNetError: 468490:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14917.172756] LNetError: 468490:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14917.182849] LNet: Removed LNI 192.168.202.104@tcp103 [14918.241066] LNet: Removed LNI 192.168.202.104@tcp104 [14919.263147] LNet: Removed LNI 192.168.202.104@tcp105 [14919.266296] LNetError: 468490:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14919.279913] LNetError: 468490:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14919.285429] LNet: Removed LNI 192.168.202.104@tcp106 [14920.351852] LNet: Removed LNI 192.168.202.104@tcp107 [14921.375170] LNet: Removed LNI 192.168.202.104@tcp108 [14922.399164] LNet: Removed LNI 192.168.202.104@tcp109 [14923.424805] LNet: Removed LNI 192.168.202.104@tcp110 [14923.426382] LNetError: 468490:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14923.444608] LNetError: 468490:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14923.463811] LNet: Removed LNI 192.168.202.104@tcp111 [14924.512099] LNet: Removed LNI 192.168.202.104@tcp112 [14925.535169] LNet: Removed LNI 192.168.202.104@tcp113 [14926.559197] LNet: Removed LNI 192.168.202.104@tcp114 [14927.583179] LNet: Removed LNI 192.168.202.104@tcp115 [14928.607287] LNet: Removed LNI 192.168.202.104@tcp116 [14929.631164] LNet: Removed LNI 192.168.202.104@tcp117 [14930.655254] LNet: Removed LNI 192.168.202.104@tcp118 [14931.679492] LNet: Removed LNI 192.168.202.104@tcp119 [14931.684949] LNetError: 468490:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14931.689600] LNetError: 468490:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14931.704994] LNet: Removed LNI 192.168.202.104@tcp120 [14932.767458] LNet: Removed LNI 192.168.202.104@tcp121 [14933.792947] LNet: Removed LNI 192.168.202.104@tcp122 [14934.815188] LNet: Removed LNI 192.168.202.104@tcp123 [14935.839252] LNet: Removed LNI 192.168.202.104@tcp124 [14936.864066] LNet: Removed LNI 192.168.202.104@tcp125 [14937.887165] LNet: Removed LNI 192.168.202.104@tcp126 [14938.911212] LNet: Removed LNI 192.168.202.104@tcp127 [14939.935202] LNet: Removed LNI 192.168.202.104@tcp128 [14940.959170] LNet: Removed LNI 192.168.202.104@tcp129 [14940.963217] LNet: Removed LNI 192.168.202.104@tcp130 [14940.971344] LNet: Removed LNI 192.168.202.104@tcp131 [14940.976536] LNet: Removed LNI 192.168.202.104@tcp132 [14940.986234] LNet: Removed LNI 192.168.202.104@tcp133 [14940.997364] LNet: Removed LNI 192.168.202.104@tcp134 [14941.019771] LNet: Removed LNI 192.168.202.104@tcp135 [14942.770548] Key type .llcrypt unregistered [14942.773371] Key type ._llcrypt unregistered [14972.940774] Key type ._llcrypt registered [14972.948499] Key type .llcrypt registered [14973.117348] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [14992.475826] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [14993.730326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14994.005036] alg: No test for adler32 (adler32-zlib) [14995.290386] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [14995.654389] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [14997.423220] Key type lgssc registered [14998.746916] Lustre: Echo OBD driver; http://www.lustre.org/ [15042.028501] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 22:16:12 (1786414572) [15062.901744] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [15066.987255] Key type lgssc unregistered [15067.412899] LNet: 472583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15067.422934] LNetError: 472583:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15067.454821] LNet: Removed LNI 192.168.202.104@tcp [15068.635787] Key type .llcrypt unregistered [15068.640878] Key type ._llcrypt unregistered [15089.678563] Key type ._llcrypt registered [15089.680428] Key type .llcrypt registered [15089.791884] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [15092.148215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15092.180443] alg: No test for adler32 (adler32-zlib) [15093.641040] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [15094.170690] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [15095.874495] Key type lgssc registered [15096.794886] Lustre: Echo OBD driver; http://www.lustre.org/ [15108.022211] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15108.055808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15109.565798] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15109.635900] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15109.839623] Lustre: lustre-MDT0000: new disk, initializing [15110.011556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15110.028755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15116.042265] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [15128.482079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15128.672972] Lustre: 473808:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15128.764221] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15128.767849] Lustre: Skipped 1 previous similar message [15128.889632] Lustre: lustre-MDT0001: new disk, initializing [15129.116452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15129.192721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15129.214502] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15135.431773] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [15148.432187] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15155.791968] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15164.298330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15164.579342] Lustre: lustre-OST0000: new disk, initializing [15164.584490] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15164.603559] Lustre: 475776:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15164.682804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15171.169590] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [15171.676920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15171.693485] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15171.865413] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15181.372724] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15185.725668] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing lnet_if_list [15191.407332] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15191.434853] LNet: Added LNI 192.168.202.104@tcp10 [8/256/0/180] [15197.543691] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15197.578968] LNet: Added LNI 192.168.202.104@tcp11 [8/256/0/180] [15203.303447] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15203.353663] LNet: Added LNI 192.168.202.104@tcp12 [8/256/0/180] [15210.890620] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15210.953237] LNet: Added LNI 192.168.202.104@tcp13 [8/256/0/180] [15217.988287] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15218.044394] LNet: Added LNI 192.168.202.104@tcp14 [8/256/0/180] [15222.631067] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15222.686848] LNet: Added LNI 192.168.202.104@tcp15 [8/256/0/180] [15228.082762] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15228.118979] LNet: Added LNI 192.168.202.104@tcp16 [8/256/0/180] [15233.529852] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15233.584952] LNet: Added LNI 192.168.202.104@tcp17 [8/256/0/180] [15239.442104] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15239.474302] LNet: Added LNI 192.168.202.104@tcp18 [8/256/0/180] [15245.397440] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15245.469243] LNet: Added LNI 192.168.202.104@tcp19 [8/256/0/180] [15251.380300] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15251.435386] LNet: Added LNI 192.168.202.104@tcp20 [8/256/0/180] [15257.411342] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15257.433559] LNet: Added LNI 192.168.202.104@tcp21 [8/256/0/180] [15264.576583] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15264.601337] LNet: Added LNI 192.168.202.104@tcp22 [8/256/0/180] [15270.516861] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15270.537601] LNet: Added LNI 192.168.202.104@tcp23 [8/256/0/180] [15277.034435] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15277.072037] LNet: Added LNI 192.168.202.104@tcp24 [8/256/0/180] [15283.779112] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15283.859572] LNet: Added LNI 192.168.202.104@tcp25 [8/256/0/180] [15290.800825] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15290.854105] LNet: Added LNI 192.168.202.104@tcp26 [8/256/0/180] [15297.919496] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15297.954445] LNet: Added LNI 192.168.202.104@tcp27 [8/256/0/180] [15304.716594] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15304.741748] LNet: Added LNI 192.168.202.104@tcp28 [8/256/0/180] [15313.226981] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15313.282110] LNet: Added LNI 192.168.202.104@tcp29 [8/256/0/180] [15321.270077] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15321.300680] LNet: Added LNI 192.168.202.104@tcp30 [8/256/0/180] [15328.787770] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15328.861545] LNet: Added LNI 192.168.202.104@tcp31 [8/256/0/180] [15337.556718] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15337.621292] LNet: Added LNI 192.168.202.104@tcp32 [8/256/0/180] [15345.577428] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15345.617412] LNet: Added LNI 192.168.202.104@tcp33 [8/256/0/180] [15353.628144] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15353.659952] LNet: Added LNI 192.168.202.104@tcp34 [8/256/0/180] [15360.960991] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15360.989606] LNet: Added LNI 192.168.202.104@tcp35 [8/256/0/180] [15368.784953] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15368.820559] LNet: Added LNI 192.168.202.104@tcp36 [8/256/0/180] [15377.433752] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15377.468348] LNet: Added LNI 192.168.202.104@tcp37 [8/256/0/180] [15385.192207] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15385.224984] LNet: Added LNI 192.168.202.104@tcp38 [8/256/0/180] [15393.809269] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15393.865375] LNet: Added LNI 192.168.202.104@tcp39 [8/256/0/180] [15402.271518] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15402.295410] LNet: Added LNI 192.168.202.104@tcp40 [8/256/0/180] [15411.448861] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15411.478151] LNet: Added LNI 192.168.202.104@tcp41 [8/256/0/180] [15420.230682] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15420.277860] LNet: Added LNI 192.168.202.104@tcp42 [8/256/0/180] [15429.254336] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15429.290071] LNet: Added LNI 192.168.202.104@tcp43 [8/256/0/180] [15438.216954] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15438.243743] LNet: Added LNI 192.168.202.104@tcp44 [8/256/0/180] [15447.383474] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15447.410771] LNet: Added LNI 192.168.202.104@tcp45 [8/256/0/180] [15456.499354] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15456.550444] LNet: Added LNI 192.168.202.104@tcp46 [8/256/0/180] [15466.107755] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15466.166324] LNet: Added LNI 192.168.202.104@tcp47 [8/256/0/180] [15478.341885] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15478.384664] LNet: Added LNI 192.168.202.104@tcp48 [8/256/0/180] [15490.914809] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15490.960578] LNet: Added LNI 192.168.202.104@tcp49 [8/256/0/180] [15503.457267] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15503.520306] LNet: Added LNI 192.168.202.104@tcp50 [8/256/0/180] [15525.345324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15525.363842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15525.383499] Lustre: Skipped 2 previous similar messages [15529.873995] Lustre: server umount lustre-MDT0000 complete [15534.361481] LustreError: 473798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786415067 with bad export cookie 6179468781153688991 [15534.365918] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15534.370689] LustreError: 473798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15534.831070] Lustre: server umount lustre-MDT0001 complete [15551.903277] Lustre: 473076:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786415068/real 1786415068] req@00000000eb90ca83 x1873191504108032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786415084 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15551.930109] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15551.946450] Lustre: Skipped 2 previous similar messages [15555.041342] Lustre: 473079:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786415072/real 1786415072] req@000000008e95632b x1873191504108160/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786415088 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15556.063313] Lustre: 473078:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786415073/real 1786415073] req@000000009c212844 x1873191504108288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786415089 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15556.520190] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [15560.095398] Lustre: 473076:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786415077/real 1786415077] req@00000000e0ea680e x1873191504108544/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786415093 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15565.281116] Lustre: 473077:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786415082/real 1786415082] req@0000000036ac4671 x1873191504108672/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786415098 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15567.122070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15580.023661] LustreError: 475769:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15580.054323] LustreError: 475769:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [15580.087310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15581.089740] LustreError: 475770:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15581.103055] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15582.125020] LustreError: 475769:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15585.776610] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [15587.305613] LustreError: 475770:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15592.423439] LustreError: 475769:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15596.159040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15596.529747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15597.552570] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15601.898162] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [15610.399666] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15616.571845] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15637.472763] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15637.479773] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15637.494104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15638.499147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15638.526811] Lustre: Skipped 1 previous similar message [15643.638774] LustreError: 475758:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15643.811737] Lustre: server umount lustre-OST0000 complete [15648.738612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15648.757335] Lustre: Skipped 1 previous similar message [15648.769224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15653.857935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15653.872304] Lustre: Skipped 2 previous similar messages [15656.333484] Lustre: server umount lustre-MDT0000 complete [15660.912265] LustreError: 473798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786415193 with bad export cookie 6179468781153691532 [15660.928046] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15660.932640] LustreError: 473798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [15661.314068] Lustre: server umount lustre-MDT0001 complete [15673.912494] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [15676.962515] Key type lgssc unregistered [15677.417269] LNet: 502829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15677.422519] LNetError: 502829:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15677.430633] LNet: Removed LNI 192.168.202.104@tcp [15677.432489] LNet: Removed LNI 192.168.202.104@tcp10 [15677.439901] LNet: Removed LNI 192.168.202.104@tcp11 [15677.447465] LNet: Removed LNI 192.168.202.104@tcp12 [15677.455808] LNet: Removed LNI 192.168.202.104@tcp13 [15677.458954] LNet: Removed LNI 192.168.202.104@tcp14 [15677.463854] LNet: Removed LNI 192.168.202.104@tcp15 [15677.465812] LNet: Removed LNI 192.168.202.104@tcp16 [15677.468254] LNet: Removed LNI 192.168.202.104@tcp17 [15677.470277] LNet: Removed LNI 192.168.202.104@tcp18 [15677.472387] LNet: Removed LNI 192.168.202.104@tcp19 [15677.474927] LNet: Removed LNI 192.168.202.104@tcp20 [15677.481673] LNet: Removed LNI 192.168.202.104@tcp21 [15677.495680] LNet: Removed LNI 192.168.202.104@tcp22 [15677.500669] LNet: Removed LNI 192.168.202.104@tcp23 [15677.510391] LNet: Removed LNI 192.168.202.104@tcp24 [15677.523838] LNet: Removed LNI 192.168.202.104@tcp25 [15677.525652] LNet: Removed LNI 192.168.202.104@tcp26 [15677.537520] LNet: Removed LNI 192.168.202.104@tcp27 [15677.539466] LNet: Removed LNI 192.168.202.104@tcp28 [15677.551752] LNet: Removed LNI 192.168.202.104@tcp29 [15677.553832] LNet: Removed LNI 192.168.202.104@tcp30 [15677.559418] LNet: Removed LNI 192.168.202.104@tcp31 [15677.564287] LNet: Removed LNI 192.168.202.104@tcp32 [15677.566718] LNet: Removed LNI 192.168.202.104@tcp33 [15677.569849] LNet: Removed LNI 192.168.202.104@tcp34 [15677.572485] LNet: Removed LNI 192.168.202.104@tcp35 [15677.577574] LNet: Removed LNI 192.168.202.104@tcp36 [15677.580367] LNet: Removed LNI 192.168.202.104@tcp37 [15677.586218] LNet: Removed LNI 192.168.202.104@tcp38 [15677.589640] LNet: Removed LNI 192.168.202.104@tcp39 [15677.593380] LNet: Removed LNI 192.168.202.104@tcp40 [15677.597231] LNet: Removed LNI 192.168.202.104@tcp41 [15677.599447] LNet: Removed LNI 192.168.202.104@tcp42 [15677.602805] LNet: Removed LNI 192.168.202.104@tcp43 [15677.609467] LNet: Removed LNI 192.168.202.104@tcp44 [15677.611286] LNet: Removed LNI 192.168.202.104@tcp45 [15677.615891] LNet: Removed LNI 192.168.202.104@tcp46 [15677.621670] LNet: Removed LNI 192.168.202.104@tcp47 [15677.625996] LNet: Removed LNI 192.168.202.104@tcp48 [15677.628299] LNet: Removed LNI 192.168.202.104@tcp49 [15677.649299] LNet: Removed LNI 192.168.202.104@tcp50 [15678.439574] Key type .llcrypt unregistered [15678.441821] Key type ._llcrypt unregistered [15700.623874] Key type ._llcrypt registered [15700.626862] Key type .llcrypt registered [15700.759842] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [15717.745277] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [15718.906438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15718.991916] alg: No test for adler32 (adler32-zlib) [15720.411719] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [15720.802392] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [15722.527169] Key type lgssc registered [15723.867441] Lustre: Echo OBD driver; http://www.lustre.org/ [15766.730094] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15766.750525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15767.908910] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15767.933282] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15767.998845] Lustre: lustre-MDT0000: new disk, initializing [15768.063178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15768.077486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15772.727384] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [15785.449183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15785.686398] Lustre: 506025:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15785.727303] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15785.729950] Lustre: Skipped 1 previous similar message [15785.785203] Lustre: lustre-MDT0001: new disk, initializing [15785.858196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15785.892050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15785.903069] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15791.643472] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [15802.793398] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15809.678476] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15818.288390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15818.620441] Lustre: *** cfs_fail_loc=90f, val=0*** [15818.745638] Lustre: lustre-OST0000: new disk, initializing [15818.749642] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15818.755847] Lustre: 508043:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15818.847099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15824.479068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15824.503871] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15824.695175] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15826.680610] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [15837.480842] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15848.423768] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15848.433191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15848.454331] Lustre: Skipped 1 previous similar message [15848.469566] Lustre: Skipped 1 previous similar message [15853.090521] Lustre: server umount lustre-OST0000 complete [15858.657694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15858.659437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15858.662941] Lustre: Skipped 1 previous similar message [15863.780186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15863.796508] Lustre: Skipped 2 previous similar messages [15864.586770] Lustre: server umount lustre-MDT0000 complete [15867.423867] LustreError: 506018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786415400 with bad export cookie 1729676365721508844 [15867.427169] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15867.437726] LustreError: 506018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15867.796456] Lustre: server umount lustre-MDT0001 complete [15880.429470] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [15884.069043] Key type lgssc unregistered [15884.444202] LNet: 509952:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15884.453445] LNetError: 509952:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15884.469961] LNet: Removed LNI 192.168.202.104@tcp [15885.287904] Key type .llcrypt unregistered [15885.291781] Key type ._llcrypt unregistered [15908.461451] Key type ._llcrypt registered [15908.467193] Key type .llcrypt registered [15908.595983] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [15909.806693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15909.920448] alg: No test for adler32 (adler32-zlib) [15911.521917] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [15911.815112] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [15913.495727] Key type lgssc registered [15914.744139] Lustre: Echo OBD driver; http://www.lustre.org/ [15923.032833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15923.078927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15924.738520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15930.057296] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [15934.945623] LustreError: 511255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15934.968290] LustreError: 511255:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [15939.592228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15940.158109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15945.858955] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [15955.333879] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15961.862621] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15969.957417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15970.321747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15976.716519] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [15984.968574] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [16004.771545] loop0: detected capacity change from 0 to 204800000 [16004.964067] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16004.978955] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16004.997918] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16012.858555] loop0: detected capacity change from 0 to 204800000 [16013.095084] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16013.113861] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16013.139432] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16019.300449] loop0: detected capacity change from 0 to 204800000 [16026.618241] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16026.982338] Lustre: fs15246-OST0000: new disk, initializing [16026.987987] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16027.010848] Lustre: 514870:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16027.087971] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16034.887605] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [16046.310499] loop1: detected capacity change from 0 to 204800000 [16053.069662] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16053.215756] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16053.270653] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16053.363514] Lustre: fs15246-MDT0000: new disk, initializing [16053.617568] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16053.682325] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16053.862944] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16054.967045] Lustre: 515993:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000008d0aede1 x1873192360660864/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786415603 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16058.974922] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [16063.083854] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16063.092938] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16063.155517] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16084.978324] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16085.002205] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16088.018683] Lustre: server umount fs15246-MDT0000 complete [16090.085225] LustreError: 513097:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16095.212580] LustreError: 511250:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16099.544536] Lustre: server umount fs15246-OST0000 complete [16112.608489] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16112.620647] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16112.641184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16113.632295] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16113.643086] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16113.663222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16115.683747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16115.694324] Lustre: Skipped 1 previous similar message [16118.604978] Lustre: server umount lustre-OST0000 complete [16124.384650] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16124.392196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16124.412188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16129.391033] Lustre: server umount lustre-MDT0000 complete [16131.041872] LustreError: 512195:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16131.080835] LustreError: 512195:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [16133.618722] LustreError: 511234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786415666 with bad export cookie 5775507356004583844 [16133.628724] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16133.632527] LustreError: 511234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16133.986181] Lustre: server umount lustre-MDT0001 complete [16145.940976] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [16149.269607] Key type lgssc unregistered [16149.656772] LNet: 519093:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16149.679039] LNetError: 519093:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [16149.707395] LNet: Removed LNI 192.168.202.104@tcp [16150.880242] Key type .llcrypt unregistered [16150.882476] Key type ._llcrypt unregistered [16170.870364] Key type ._llcrypt registered [16170.872447] Key type .llcrypt registered [16170.979962] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [16172.612938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16172.649979] alg: No test for adler32 (adler32-zlib) [16173.686806] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [16173.982578] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [16175.663365] Key type lgssc registered [16177.335149] Lustre: Echo OBD driver; http://www.lustre.org/ [16221.368887] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [16233.011775] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [16274.914360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16274.944415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16276.341891] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16276.394232] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16276.456162] Lustre: lustre-MDT0000: new disk, initializing [16276.609470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16276.623939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16282.063296] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [16294.037280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16294.209539] Lustre: 523562:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16294.309377] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16294.311846] Lustre: Skipped 1 previous similar message [16294.455105] Lustre: lustre-MDT0001: new disk, initializing [16294.585025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16294.645797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16294.667864] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16300.736138] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [16311.270563] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16317.515218] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [16325.764615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16326.213714] Lustre: lustre-OST0000: new disk, initializing [16326.221627] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16326.236581] Lustre: 525531:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16326.370873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16333.873603] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [16334.456777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16334.483717] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16334.617708] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16345.035268] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [16350.189855] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16350.504432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16356.206791] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16356.555426] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16360.420079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16360.423051] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16360.429625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16360.430396] Lustre: Skipped 1 previous similar message [16360.447719] Lustre: Skipped 1 previous similar message [16365.551107] Lustre: server umount lustre-OST0000 complete [16370.656286] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16370.660973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16370.672545] Lustre: Skipped 1 previous similar message [16370.686661] Lustre: Skipped 1 previous similar message [16375.776270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16375.780626] Lustre: Skipped 1 previous similar message [16376.731430] Lustre: server umount lustre-MDT0000 complete [16381.080945] LustreError: 523555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786415914 with bad export cookie 4863725937736334667 [16381.091515] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16381.479183] Lustre: server umount lustre-MDT0001 complete [16409.255306] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [16421.331960] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [16456.170643] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16209 sec ========= 22:39:47 (1786415987) [16458.139542] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:39:49 (1786415989) === [16461.661469] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:39:52 (1786415992) === [16504.243743] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [16507.353603] Key type lgssc unregistered [16507.756104] LNet: 532178:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16507.760311] LNetError: 532178:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [16507.774724] LNet: Removed LNI 192.168.202.104@tcp [16508.741620] Key type .llcrypt unregistered [16508.743368] Key type ._llcrypt unregistered