[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 478275513 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2576MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001020] APIC: Switch to symmetric I/O mode setup [ 0.002410] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009122] LSM: Security Framework initializing [ 0.010050] Yama: becoming mindful. [ 0.011038] SELinux: Initializing. [ 0.012071] *** VALIDATE selinux *** [ 0.020707] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024766] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025138] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027091] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029095] *** VALIDATE tmpfs *** [ 0.031284] *** VALIDATE proc *** [ 0.032233] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.034249] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036153] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038029] Spectre V2 : User space: Vulnerable [ 0.039008] Speculative Store Bypass: Vulnerable [ 0.042158] debug: unmapping init [mem 0xffffffffb1c59000-0xffffffffb1c60fff] [ 0.044163] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045656] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046020] ... version: 2 [ 0.047009] ... bit width: 48 [ 0.048009] ... generic registers: 4 [ 0.049008] ... value mask: 0000ffffffffffff [ 0.050016] ... max period: 00007fffffffffff [ 0.051012] ... fixed-purpose events: 3 [ 0.052010] ... event mask: 000000070000000f [ 0.053289] rcu: Hierarchical SRCU implementation. [ 0.055434] smp: Bringing up secondary CPUs ... [ 0.056561] x86: Booting SMP configuration: [ 0.057028] .... node #0, CPUs: #1 #2 #3 [ 0.060700] smp: Brought up 1 node, 4 CPUs [ 0.062011] smpboot: Max logical packages: 1 [ 0.063016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.151735] node 0 deferred pages initialised in 86ms [ 0.155097] devtmpfs: initialized [ 0.156285] x86/mm: Memory block size: 128MB [ 0.160000] gcov: version magic: 0x41383552 [ 0.163410] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.167091] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.169330] pinctrl core: initialized pinctrl subsystem [ 0.171181] [ 0.171796] ************************************************************* [ 0.174013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.177015] ** ** [ 0.179011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.181014] ** ** [ 0.184012] ** This means that this kernel is built to expose internal ** [ 0.186011] ** IOMMU data structures, which may compromise security on ** [ 0.188010] ** your system. ** [ 0.190013] ** ** [ 0.193021] ** If you see this message and you are not debugging the ** [ 0.195011] ** kernel, report this immediately to your vendor! ** [ 0.197054] ** ** [ 0.200013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.202012] ************************************************************* [ 0.205715] NET: Registered protocol family 16 [ 0.207468] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.209071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.213063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.217096] cpuidle: using governor menu [ 0.218549] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.220610] PCI: Using configuration type 1 for base access [ 0.222140] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.231115] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.236019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.240057] cryptd: max_cpu_qlen set to 1000 [ 0.242172] ACPI: Added _OSI(Module Device) [ 0.244014] ACPI: Added _OSI(Processor Device) [ 0.246015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.248014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.252855] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.259572] ACPI: Interpreter enabled [ 0.261082] ACPI: PM: (supports S0 S3 S4 S5) [ 0.263013] ACPI: Using IOAPIC for interrupt routing [ 0.264121] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.268364] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.278934] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.281043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.283014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.287074] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.291179] acpiphp: Slot [2] registered [ 0.293097] acpiphp: Slot [5] registered [ 0.294127] acpiphp: Slot [6] registered [ 0.296079] acpiphp: Slot [7] registered [ 0.297135] acpiphp: Slot [8] registered [ 0.299143] acpiphp: Slot [9] registered [ 0.301119] acpiphp: Slot [10] registered [ 0.302149] acpiphp: Slot [3] registered [ 0.304096] acpiphp: Slot [4] registered [ 0.305104] acpiphp: Slot [11] registered [ 0.307110] acpiphp: Slot [12] registered [ 0.309098] acpiphp: Slot [13] registered [ 0.310098] acpiphp: Slot [14] registered [ 0.312104] acpiphp: Slot [15] registered [ 0.313121] acpiphp: Slot [16] registered [ 0.315092] acpiphp: Slot [17] registered [ 0.316130] acpiphp: Slot [18] registered [ 0.317092] acpiphp: Slot [19] registered [ 0.319082] acpiphp: Slot [20] registered [ 0.320082] acpiphp: Slot [21] registered [ 0.321083] acpiphp: Slot [22] registered [ 0.322085] acpiphp: Slot [23] registered [ 0.323107] acpiphp: Slot [24] registered [ 0.324085] acpiphp: Slot [25] registered [ 0.325085] acpiphp: Slot [26] registered [ 0.327083] acpiphp: Slot [27] registered [ 0.328091] acpiphp: Slot [28] registered [ 0.330129] acpiphp: Slot [29] registered [ 0.332118] acpiphp: Slot [30] registered [ 0.333080] acpiphp: Slot [31] registered [ 0.335052] PCI host bridge to bus 0000:00 [ 0.336023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.339027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.341023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.345025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.347023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.350020] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.351121] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.353026] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.357000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.366023] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.371507] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.374017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.376020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.378014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.381116] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.382913] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.385039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.387766] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.392014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.405016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.410021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.414784] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.422012] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.425000] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.443017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.455168] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.465013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.474015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.495014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.508577] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.516015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.524018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.547021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.556985] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.568015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.577016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.595015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.606000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.618018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.627015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.640016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.648652] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.657014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.666017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.684016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.703149] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.706479] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.709461] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.712427] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.715295] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.720082] iommu: Default domain type: Passthrough [ 0.722396] SCSI subsystem initialized [ 0.724155] ACPI: bus type USB registered [ 0.726135] usbcore: registered new interface driver usbfs [ 0.728119] usbcore: registered new interface driver hub [ 0.730143] usbcore: registered new device driver usb [ 0.733177] pps_core: LinuxPPS API ver. 1 registered [ 0.735012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.739061] PTP clock support registered [ 0.743068] EDAC MC: Ver: 3.0.0 [ 0.745186] PCI: Using ACPI for IRQ routing [ 0.746957] NetLabel: Initializing [ 0.747009] NetLabel: domain hash size = 128 [ 0.748009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.749123] NetLabel: unlabeled traffic allowed by default [ 0.751369] vgaarb: loaded [ 0.753220] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.754014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.762117] clocksource: Switched to clocksource kvm-clock [ 0.864369] VFS: Disk quotas dquot_6.6.0 [ 0.866598] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.869214] *** VALIDATE ramfs *** [ 0.870399] *** VALIDATE hugetlbfs *** [ 0.872027] pnp: PnP ACPI init [ 0.874435] pnp: PnP ACPI: found 6 devices [ 0.902366] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.906933] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.909525] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.912481] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.915694] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.918719] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.922881] NET: Registered protocol family 2 [ 0.925678] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.930704] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.935191] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.941026] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.944392] TCP: Hash tables configured (established 65536 bind 65536) [ 0.947581] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.950314] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.952717] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.955318] NET: Registered protocol family 1 [ 0.957421] RPC: Registered named UNIX socket transport module. [ 0.959662] RPC: Registered udp transport module. [ 0.961420] RPC: Registered tcp transport module. [ 0.963417] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.966259] NET: Registered protocol family 44 [ 0.968166] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.970634] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.973106] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.975626] PCI: CLS 0 bytes, default 64 [ 0.977629] Unpacking initramfs... [ 2.370778] debug: unmapping init [mem 0xffff9fc3fcc54000-0xffff9fc3fffbffff] [ 2.375580] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.378056] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.383306] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.880745] Initialise system trusted keyrings [ 2.883400] Key type blacklist registered [ 2.885703] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.895246] zbud: loaded [ 2.898464] *** VALIDATE nfs *** [ 2.899777] *** VALIDATE nfs4 *** [ 2.901568] pstore: using deflate compression [ 2.906313] Platform Keyring initialized [ 3.041703] NET: Registered protocol family 38 [ 3.045465] Key type asymmetric registered [ 3.047339] Asymmetric key parser 'x509' registered [ 3.049680] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.053179] io scheduler mq-deadline registered [ 3.055199] io scheduler kyber registered [ 3.057246] io scheduler bfq registered [ 3.060138] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.063452] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.066811] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.070167] ACPI: Power Button [PWRF] [ 3.076445] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.085313] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.098495] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.105194] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.118058] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.146752] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.178935] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.184615] Non-volatile memory driver v1.3 [ 3.186729] Linux agpgart interface v0.103 [ 3.222210] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.225608] vda: detected capacity change from 0 to 76677120 [ 3.247228] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.250348] vdb: detected capacity change from 0 to 1073741824 [ 3.270370] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.273815] vdc: detected capacity change from 0 to 2621440000 [ 3.289278] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.291948] vdd: detected capacity change from 0 to 2621440000 [ 3.308086] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.311205] vde: detected capacity change from 0 to 4294967296 [ 3.327309] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.330404] vdf: detected capacity change from 0 to 4294967296 [ 3.341727] libphy: Fixed MDIO Bus: probed [ 3.347245] usbcore: registered new interface driver usbserial_generic [ 3.350233] usbserial: USB Serial support registered for generic [ 3.353105] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.357564] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.359590] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.362206] mousedev: PS/2 mouse device common for all mice [ 3.365280] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.366760] rtc_cmos 00:05: RTC can wake from S4 [ 3.371750] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.372230] rtc_cmos 00:05: registered as rtc0 [ 3.375586] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.377330] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.383522] intel_pstate: CPU model not supported [ 3.387067] hid: raw HID events driver (C) Jiri Kosina [ 3.389752] usbcore: registered new interface driver usbhid [ 3.392314] usbhid: USB HID core driver [ 3.394287] drop_monitor: Initializing network drop monitor service [ 3.397511] Initializing XFRM netlink socket [ 3.399785] NET: Registered protocol family 10 [ 3.403525] Segment Routing with IPv6 [ 3.404864] NET: Registered protocol family 17 [ 3.406877] mpls_gso: MPLS GSO support [ 3.413420] RAS: Correctable Errors collector initialized. [ 3.415930] AVX version of gcm_enc/dec engaged. [ 3.417673] AES CTR mode by8 optimization enabled [ 3.492188] sched_clock: Marking stable (3492155915, 0)->(4408875781, -916719866) [ 3.496328] registered taskstats version 1 [ 3.498350] Loading compiled-in X.509 certificates [ 3.500199] zswap: loaded using pool lzo/zbud [ 3.523887] Key type big_key registered [ 3.535895] Key type encrypted registered [ 3.537318] ima: No TPM chip found, activating TPM-bypass! [ 3.539178] ima: Allocated hash algorithm: sha1 [ 3.540467] ima: No architecture policies found [ 3.542036] evm: Initialising EVM extended attributes: [ 3.543572] evm: security.selinux [ 3.544553] evm: security.ima [ 3.545480] evm: security.capability [ 3.546585] evm: HMAC attrs: 0x1 [ 3.549091] rtc_cmos 00:05: setting system clock to 2026-09-03 12:44:19 UTC (1788439459) [ 3.556383] debug: unmapping init [mem 0xffffffffb2c03000-0xffffffffb2dfffff] [ 3.560302] debug: unmapping init [mem 0xffffffffb1982000-0xffffffffb1c58fff] [ 3.569086] Write protecting the kernel read-only data: 28672k [ 3.573187] debug: unmapping init [mem 0xffffffffb0003000-0xffffffffb01fffff] [ 3.576869] debug: unmapping init [mem 0xffffffffb0914000-0xffffffffb09fffff] [ 3.611810] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.620759] systemd[1]: Detected virtualization kvm. [ 3.622675] systemd[1]: Detected architecture x86-64. [ 3.624154] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.648392] systemd[1]: No hostname configured. [ 3.651190] systemd[1]: Set hostname to . [ 3.653439] random: systemd: uninitialized urandom read (16 bytes read) [ 3.655268] systemd[1]: Initializing machine ID from random generator. [ 3.790871] random: systemd: uninitialized urandom read (16 bytes read) [ 3.793291] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.798230] random: systemd: uninitialized urandom read (16 bytes read) [ 3.801630] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.805733] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.476239] device-mapper: uevent: version 1.0.3 [ 4.478524] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.178907] random: fast init done [ 5.208611] virtio_net virtio0 ens2: renamed from eth0 [ 5.239908] scsi host0: ata_piix [ 5.354425] scsi host1: ata_piix [ 5.356652] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.360103] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.582163] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.051928] random: crng init done [ 10.053296] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.452696] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.679488] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.926636] SELinux: Disabled at runtime. [ 11.992672] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.999368] systemd[1]: Detected virtualization kvm. [ 12.000734] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.567593] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.571833] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.581559] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.588427] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.592425] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.600600] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.606423] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ 12.658336] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ 13.032942] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.415890] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.508732] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.601642] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.614243] EDAC sbridge: Ver: 1.1.2 [ 15.288753] Key type dns_resolver registered [ 15.585606] NFS: Registering the id_resolver key type [ 15.587557] Key type id_resolver registered [ 15.589102] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg339-server login: [ 42.351593] libcfs: loading out-of-tree module taints kernel. [ 42.371944] Key type ._llcrypt registered [ 42.373901] Key type .llcrypt registered [ 42.430772] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 51.925616] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 52.696323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 52.709361] alg: No test for adler32 (adler32-zlib) [ 53.766478] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 54.126198] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 55.759188] Key type lgssc registered [ 56.603697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 71.791005] hrtimer: interrupt took 27996877 ns [ 77.800371] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 121.283456] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 136.145493] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 136.217320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 137.513623] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 137.548594] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 137.677547] Lustre: lustre-MDT0000: new disk, initializing [ 137.798867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 137.820476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 143.291261] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 159.021648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 159.141252] Lustre: 6483:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 159.168785] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 159.173446] Lustre: Skipped 1 previous similar message [ 159.293227] Lustre: lustre-MDT0001: new disk, initializing [ 159.413973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 159.458966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 159.484395] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 164.478889] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 169.399780] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 179.299203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 179.644993] Lustre: lustre-OST0000: new disk, initializing [ 179.655132] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 179.665589] Lustre: 8421:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 179.748572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 184.345614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 184.362455] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 184.467018] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 186.317623] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 201.242044] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 201.457590] Lustre: lustre-OST0001: new disk, initializing [ 201.461068] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 201.467382] Lustre: 9493:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 201.516686] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 207.911794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 207.922741] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 207.972763] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 208.285674] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.035977] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 228.828329] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 236.292849] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing check_logdir /tmp/testlogs/ [ 241.718362] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing yml_node [ 246.567495] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 249.603678] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 252.117117] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 253.924329] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 08:48:28 EDT 2026 [ 271.784475] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 273.411699] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 289.769268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 289.773177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 289.781060] Lustre: Skipped 3 previous similar messages [ 289.796456] Lustre: Skipped 3 previous similar messages [ 294.882202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 294.901338] Lustre: Skipped 3 previous similar messages [ 295.664495] Lustre: server umount lustre-MDT0000 complete [ 300.206505] LustreError: 10367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439756 with bad export cookie 4897631696686796590 [ 300.206816] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 300.232047] LustreError: 10367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 303.009600] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 303.032776] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 303.037033] Lustre: Skipped 1 previous similar message [ 305.056030] LustreError: 7708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 305.056917] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 305.076576] LustreError: 7708:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 305.089302] Lustre: Skipped 1 previous similar message [ 306.634656] Lustre: server umount lustre-MDT0001 complete [ 310.730276] Lustre: server umount lustre-OST0000 complete [ 314.604502] Lustre: server umount lustre-OST0001 complete [ 321.071420] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 330.478878] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 370.584556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 370.997332] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 371.038865] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 371.184481] Lustre: lustre-MDT0000: new disk, initializing [ 371.259330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 371.282991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 376.151177] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 386.887817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 386.951410] Lustre: 15046:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 386.993492] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 386.996973] Lustre: Skipped 1 previous similar message [ 387.064307] Lustre: lustre-MDT0001: new disk, initializing [ 387.159969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 387.216544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 387.234853] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 392.357440] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 403.507934] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 411.109470] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 419.118869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 419.624599] Lustre: lustre-OST0000: new disk, initializing [ 419.629734] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 419.634245] Lustre: 17014:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 419.756112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 421.586199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 421.600985] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 421.759487] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 427.018972] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 436.630371] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 442.458932] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 442.727465] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 447.866587] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 448.104554] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 452.576037] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 452.582563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 452.586903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 452.605756] Lustre: Skipped 2 previous similar messages [ 452.629021] Lustre: Skipped 1 previous similar message [ 462.820596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 462.837151] Lustre: Skipped 3 previous similar messages [ 464.351221] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 464.624507] Lustre: server umount lustre-OST0000 complete [ 468.961095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 468.976058] LustreError: Skipped 1 previous similar message [ 468.981508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 473.064752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 474.852990] Lustre: server umount lustre-MDT0000 complete [ 478.177421] LustreError: 15058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 478.204348] LustreError: 15058:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 478.237232] LustreError: 15842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439934 with bad export cookie 4897631696686798123 [ 478.242732] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 478.249877] LustreError: 15842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 478.819357] Lustre: server umount lustre-MDT0001 complete [ 482.849248] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 08:52:17 (1788439937) [ 485.887963] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 488.387430] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 08:52:22 (1788439942) [ 495.840846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 496.646041] LustreError: 19426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 496.810675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 501.506484] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 502.247960] LustreError: 19427:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 506.361377] LustreError: 19426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 511.569800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 517.684525] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 525.684830] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 533.116046] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 541.447711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 549.295650] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 558.042134] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 582.113330] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 582.124266] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 582.141552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 582.141560] Lustre: Skipped 3 previous similar messages [ 585.190526] Lustre: server umount lustre-OST0000 complete [ 596.506215] Lustre: server umount lustre-MDT0000 complete [ 597.476692] LustreError: 20154:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 597.506905] LustreError: 20154:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 600.180064] LustreError: 19407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440056 with bad export cookie 4897631696686799005 [ 600.184222] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 600.195875] LustreError: 19407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 600.371940] Lustre: server umount lustre-MDT0001 complete [ 611.791359] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 614.862544] Key type lgssc unregistered [ 615.264415] LNet: 22981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 615.271915] LNetError: 22981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 615.301750] LNet: Removed LNI 192.168.203.139@tcp [ 616.412366] Key type .llcrypt unregistered [ 616.419795] Key type ._llcrypt unregistered [ 639.589906] Key type ._llcrypt registered [ 639.593231] Key type .llcrypt registered [ 639.752962] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 640.855561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 641.059975] alg: No test for adler32 (adler32-zlib) [ 642.135160] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 642.383260] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 644.135156] Key type lgssc registered [ 645.456234] Lustre: Echo OBD driver; http://www.lustre.org/ [ 654.246238] loop: module loaded [ 654.301817] loop0: detected capacity change from 0 to 204800000 [ 654.591387] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 654.621273] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 654.980161] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 661.567988] loop0: detected capacity change from 0 to 204800000 [ 661.826640] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 661.839256] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 661.855329] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 667.756520] loop0: detected capacity change from 0 to 204800000 [ 674.029634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 674.066739] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 675.432403] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 675.497627] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 675.662728] Lustre: lustre-MDT0000: new disk, initializing [ 675.807263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 675.853851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 681.430384] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 691.430169] loop1: detected capacity change from 0 to 204800000 [ 697.653357] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 698.034671] Lustre: lustre-OST07c6: new disk, initializing [ 698.040784] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 698.043923] Lustre: Skipped 1 previous similar message [ 698.057845] Lustre: 26164:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 698.151793] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 704.645462] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 708.278363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 708.291778] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 708.348489] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 710.300960] Lustre: 26172:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 710.322579] Lustre: 26172:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 710.334058] Lustre: 26172:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 710.343229] Lustre: 26172:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 710.359346] Lustre: 26172:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 710.364745] Lustre: 26172:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 725.595488] Lustre: 25151:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 725.609865] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 725.615061] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 725.622220] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 725.629642] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 725.638559] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 730.737418] Lustre: 25151:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 730.749538] Lustre: 25151:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 730.755239] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 730.761241] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 730.769464] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 730.778581] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 730.786826] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 730.798455] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 730.805154] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 730.815440] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 730.822031] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 730.827066] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 734.042479] Lustre: 25152:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 734.052184] Lustre: 25152:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 734.057763] Lustre: 25152:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 734.062324] Lustre: 25152:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 734.074340] Lustre: 25152:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 734.087208] Lustre: 25152:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 734.095918] Lustre: 25152:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 734.102272] Lustre: 25152:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 734.106944] Lustre: 25152:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 734.112038] Lustre: 25152:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 734.120194] Lustre: 25152:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 734.132879] Lustre: 25152:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 739.138033] Lustre: 25153:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 739.152396] Lustre: 25153:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 739.163852] Lustre: 25153:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 739.175844] Lustre: 25153:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 739.182406] Lustre: 25153:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 739.202130] Lustre: 25153:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 739.207434] Lustre: 25153:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 739.213111] Lustre: 25153:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 739.217626] Lustre: 25153:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 739.222863] Lustre: 25153:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 739.229190] Lustre: 25153:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 739.233461] Lustre: 25153:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 747.463410] Lustre: 25152:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 747.485303] Lustre: 25152:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 747.506462] Lustre: 25152:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 747.532245] Lustre: 25152:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 747.545319] Lustre: 25152:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 747.558300] Lustre: 25152:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 747.573410] Lustre: 25152:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 747.589020] Lustre: 25152:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 747.597814] Lustre: 25152:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 747.605461] Lustre: 25152:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 747.611896] Lustre: 25152:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 747.617973] Lustre: 25152:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 764.563404] Lustre: 25152:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 764.580487] Lustre: 25152:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 7 previous similar messages [ 764.590953] Lustre: 25152:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 764.598902] Lustre: 25152:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 764.604370] Lustre: 25152:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 764.609889] Lustre: 25152:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 764.614065] Lustre: 25152:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 764.619440] Lustre: 25152:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 764.627250] Lustre: 25152:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 764.633444] Lustre: 25152:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 764.638964] Lustre: 25152:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 764.644422] Lustre: 25152:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 803.298498] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 803.328383] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 804.288631] Lustre: server umount lustre-OST07c6 complete [ 814.594831] Lustre: server umount lustre-MDT0000 complete [ 832.899822] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 836.066163] Key type lgssc unregistered [ 836.383454] LNet: 28720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 836.397194] LNetError: 28720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 837.485617] LNet: Removed LNI 192.168.203.139@tcp [ 838.359581] Key type .llcrypt unregistered [ 838.363845] Key type ._llcrypt unregistered [ 859.033857] Key type ._llcrypt registered [ 859.039855] Key type .llcrypt registered [ 859.133098] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 860.343103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 860.523960] alg: No test for adler32 (adler32-zlib) [ 861.613559] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 861.898097] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 863.551138] Key type lgssc registered [ 864.889367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 874.215796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 874.247245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 875.954278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 881.366720] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 885.242698] LustreError: 29986:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 885.272361] LustreError: 29986:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 890.344995] LustreError: 29987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 890.365855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 890.768373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 895.450686] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 903.809323] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 909.656100] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 917.317800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 917.695972] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 920.777189] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 924.542938] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 932.433271] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 951.783798] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 951.794910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 951.817554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 955.876859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 955.878963] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 955.879230] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 957.587605] Lustre: server umount lustre-OST0000 complete [ 962.529647] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 962.541821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 962.557683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 962.565266] Lustre: Skipped 1 previous similar message [ 966.119941] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 966.125788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 966.136754] Lustre: Skipped 1 previous similar message [ 967.746838] Lustre: server umount lustre-MDT0000 complete [ 971.241645] LustreError: 30991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 971.260277] LustreError: 30991:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 971.363398] LustreError: 32978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440427 with bad export cookie 12708809668301355746 [ 971.373547] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 971.380721] LustreError: 32978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 972.019674] Lustre: server umount lustre-MDT0001 complete [ 983.470703] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 986.103408] Key type lgssc unregistered [ 986.428612] LNet: 33729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 986.438541] LNetError: 33729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 986.467930] LNet: Removed LNI 192.168.203.139@tcp [ 987.383073] Key type .llcrypt unregistered [ 987.386812] Key type ._llcrypt unregistered [ 1006.807347] Key type ._llcrypt registered [ 1006.812384] Key type .llcrypt registered [ 1006.970625] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1008.573715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1008.672579] alg: No test for adler32 (adler32-zlib) [ 1009.711850] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1009.972823] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1011.651479] Key type lgssc registered [ 1012.921927] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1020.911858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1020.943673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1022.485745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1027.513970] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1032.674322] LustreError: 35010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1032.692990] LustreError: 35010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1036.659746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1037.032588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1042.033813] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1050.585798] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1057.396386] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1065.247983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1065.852354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1067.949966] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1072.949473] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1082.496452] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1093.603235] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1093.622976] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1093.643454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1096.673939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1096.685926] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1096.686934] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1099.002639] Lustre: server umount lustre-OST0000 complete [ 1103.842226] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1103.856802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1103.884835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1103.888961] Lustre: Skipped 1 previous similar message [ 1106.913914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1106.914975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1106.951811] Lustre: Skipped 1 previous similar message [ 1112.032659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1112.048875] Lustre: Skipped 1 previous similar message [ 1117.151128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1117.333282] Lustre: server umount lustre-MDT0000 complete [ 1121.236898] LustreError: 34989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440577 with bad export cookie 15680062723172349439 [ 1121.241807] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1121.256206] LustreError: 34989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1121.633874] Lustre: server umount lustre-MDT0001 complete [ 1132.943993] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1135.883639] Key type lgssc unregistered [ 1136.205809] LNet: 38546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1136.215745] LNetError: 38546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1136.228720] LNet: Removed LNI 192.168.203.139@tcp [ 1137.146514] Key type .llcrypt unregistered [ 1137.149884] Key type ._llcrypt unregistered [ 1161.876196] Key type ._llcrypt registered [ 1161.882454] Key type .llcrypt registered [ 1162.021577] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1163.584876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1163.640401] alg: No test for adler32 (adler32-zlib) [ 1164.787399] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1165.107627] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1166.849064] Key type lgssc registered [ 1168.004286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1176.397718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1176.435128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1177.978276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1183.000394] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1187.299307] LustreError: 39961:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1187.312880] LustreError: 39961:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1192.419558] LustreError: 39960:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1193.017161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1193.515317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1199.279907] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1208.722309] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1215.674476] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1223.244932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1223.769647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1230.799578] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1239.418325] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1247.347180] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1256.568941] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.39@tcp (stopping) [ 1257.952800] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1257.970862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1257.978976] Lustre: Skipped 2 previous similar messages [ 1257.983298] Lustre: Skipped 1 previous similar message [ 1258.398590] Lustre: server umount lustre-MDT0000 complete [ 1261.658606] LustreError: 39956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1262.355581] LustreError: 39940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440718 with bad export cookie 15918881422770760336 [ 1262.360266] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1262.379726] LustreError: 39940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1263.087629] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1263.115581] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1263.125757] Lustre: Skipped 1 previous similar message [ 1266.794368] LustreError: 39955:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1266.810631] LustreError: 39955:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1266.823807] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.39@tcp (stopping) [ 1268.716452] Lustre: server umount lustre-MDT0001 complete [ 1289.806308] Lustre: server umount lustre-OST0000 complete [ 1311.555734] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1315.308834] Key type lgssc unregistered [ 1315.601140] LNet: 43854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1315.605587] LNetError: 43854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1315.614891] LNet: Removed LNI 192.168.203.139@tcp [ 1316.466445] Key type .llcrypt unregistered [ 1316.471191] Key type ._llcrypt unregistered [ 1343.592596] Key type ._llcrypt registered [ 1343.598338] Key type .llcrypt registered [ 1343.705423] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1344.609665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1344.662672] alg: No test for adler32 (adler32-zlib) [ 1345.908804] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1346.277438] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1347.967272] Key type lgssc registered [ 1349.421942] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1356.586898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1356.625815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1358.029737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1362.289731] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1367.526664] LustreError: 45268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1367.543132] LustreError: 45268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1372.045140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1372.299467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1377.692960] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1386.233650] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1392.324353] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1399.135142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1399.540344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1402.615326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1406.038159] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1413.706695] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1423.341842] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1423.345929] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1423.357591] LustreError: Skipped 1 previous similar message [ 1423.364681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1423.367583] Lustre: Skipped 1 previous similar message [ 1423.389714] Lustre: Skipped 1 previous similar message [ 1425.014513] Lustre: server umount lustre-OST0000 complete [ 1442.701758] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1442.709196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1443.745283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1443.755750] Lustre: Skipped 2 previous similar messages [ 1448.930418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1448.939666] Lustre: Skipped 1 previous similar message [ 1449.047076] Lustre: server umount lustre-MDT0000 complete [ 1452.352613] LustreError: 45249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440908 with bad export cookie 7847410815023710511 [ 1452.361106] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1452.366936] LustreError: 45249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1458.786978] Lustre: server umount lustre-MDT0001 complete [ 1471.670726] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1474.380693] Key type lgssc unregistered [ 1474.649807] LNet: 48978:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1474.655558] LNetError: 48978:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1474.668973] LNet: Removed LNI 192.168.203.139@tcp [ 1475.372356] Key type .llcrypt unregistered [ 1475.374326] Key type ._llcrypt unregistered [ 1500.527197] Key type ._llcrypt registered [ 1500.533161] Key type .llcrypt registered [ 1500.679970] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1501.828194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1501.867436] alg: No test for adler32 (adler32-zlib) [ 1502.932400] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1503.151695] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1504.855160] Key type lgssc registered [ 1505.718861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1514.680644] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1514.700821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1516.344778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1521.317391] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1526.756926] LustreError: 50390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1526.778813] LustreError: 50390:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1530.386616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1530.772903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1535.765277] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1544.150522] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1549.585257] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1556.375794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1556.833970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1561.986307] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1563.461849] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1570.950405] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1580.489339] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1584.378333] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1596.821568] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-03 9h10m50s [ 1598.407731] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1600.994547] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1600.998582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1601.015360] Lustre: Skipped 1 previous similar message [ 1601.026699] LustreError: 49700:0:(import.c:761:ptlrpc_connect_import_locked()) already connecting [ 1602.091159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1602.095595] Lustre: Skipped 4 previous similar messages [ 1604.250951] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.39@tcp (stopping) [ 1606.504189] Lustre: server umount lustre-MDT0000 complete [ 1608.164814] LustreError: 50389:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1608.196208] LustreError: 50389:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1609.709231] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1611.352423] LustreError: 50386:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1614.313553] LustreError: 51384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1614.345247] LustreError: 51384:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1616.614671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1616.849447] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1617.104349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1617.190478] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1618.100504] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1621.727700] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1625.060732] Lustre: DEBUG MARKER: Wait for df (36670) ... [ 1626.495443] Lustre: DEBUG MARKER: done [ 1631.720283] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1631.737663] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1631.752798] Lustre: Skipped 1 previous similar message [ 1631.758688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1631.768234] Lustre: Skipped 1 previous similar message [ 1632.736767] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1632.745532] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1637.102135] Lustre: server umount lustre-OST0000 complete [ 1644.006496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1644.011282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1644.028804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1644.035714] Lustre: Skipped 4 previous similar messages [ 1646.709969] Lustre: server umount lustre-MDT0000 complete [ 1650.015181] LustreError: 52234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441105 with bad export cookie 16213465561346857815 [ 1650.016181] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1650.021910] LustreError: 52234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1650.207170] Lustre: server umount lustre-MDT0001 complete [ 1660.925785] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1663.814467] Key type lgssc unregistered [ 1664.081691] LNet: 55321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1664.091804] LNetError: 55321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1664.114336] LNet: Removed LNI 192.168.203.139@tcp [ 1665.077964] Key type .llcrypt unregistered [ 1665.080946] Key type ._llcrypt unregistered [ 1709.455302] Key type ._llcrypt registered [ 1709.458448] Key type .llcrypt registered [ 1709.596939] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1710.700249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1710.718776] alg: No test for adler32 (adler32-zlib) [ 1711.838838] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1712.110120] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1713.776364] Key type lgssc registered [ 1714.937734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1722.883581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1722.918026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1724.183534] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1724.235855] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1724.412277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1728.523796] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1739.856656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1740.083217] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1740.146041] Lustre: 57529:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1740.576318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1747.395188] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1756.037685] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1763.360309] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1770.868213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1771.160043] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1771.595349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1779.455457] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1781.105395] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1792.387113] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1799.087637] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-03 9h14m13s [ 1800.640693] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1808.147570] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1811.169735] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1811.268463] LustreError: 57537:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1831.591041] Lustre: lustre-MDT0000: Client 390656c2-c485-4c2d-8151-2e231dacba88 (at 192.168.203.39@tcp) reconnecting [ 1851.271237] LustreError: 57537:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1853.677115] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1856.342746] Lustre: DEBUG MARKER: done [ 1866.726904] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1866.730058] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1866.773988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1866.779989] Lustre: Skipped 1 previous similar message [ 1868.808441] Lustre: server umount lustre-OST0000 complete [ 1873.894383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1873.905969] LustreError: Skipped 1 previous similar message [ 1873.910477] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1873.928119] Lustre: Skipped 1 previous similar message [ 1873.949396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1874.980710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1874.982972] Lustre: Skipped 1 previous similar message [ 1878.055439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1878.064169] Lustre: Skipped 1 previous similar message [ 1884.131239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1884.134585] Lustre: Skipped 1 previous similar message [ 1887.199815] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1887.429287] Lustre: server umount lustre-MDT0000 complete [ 1891.387610] LustreError: 58273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441347 with bad export cookie 128220457557534099 [ 1891.398937] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1891.401134] LustreError: 58273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1891.881964] Lustre: server umount lustre-MDT0001 complete [ 1904.059298] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1907.207601] Key type lgssc unregistered [ 1907.542075] LNet: 61864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1907.552169] LNetError: 61864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1907.573139] LNet: Removed LNI 192.168.203.139@tcp [ 1908.322335] Key type .llcrypt unregistered [ 1908.325857] Key type ._llcrypt unregistered [ 1950.298525] Key type ._llcrypt registered [ 1950.300268] Key type .llcrypt registered [ 1950.443676] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1951.694839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1951.755260] alg: No test for adler32 (adler32-zlib) [ 1952.813624] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1953.020281] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1954.680660] Key type lgssc registered [ 1955.918552] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1959.142517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1959.174691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1960.597462] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1960.614847] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1960.748411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1961.694088] Lustre: Failing over lustre-MDT0000 [ 1961.916356] Lustre: server umount lustre-MDT0000 complete [ 1969.994613] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:17:03 (1788441423) [ 1978.144275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1978.645683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1984.184869] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1992.333669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1992.399298] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1992.454021] Lustre: 64655:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1992.734804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1997.380994] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2005.584760] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2011.291368] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2018.198984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2018.433631] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2018.666069] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2024.662726] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2025.601543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2031.880179] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2037.365218] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2046.434303] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2046.436838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2046.443295] Lustre: Skipped 1 previous similar message [ 2050.898895] Lustre: server umount lustre-MDT0000 complete [ 2054.901392] LustreError: 64648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441510 with bad export cookie 6767185855287412540 [ 2054.905413] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2054.917855] LustreError: 64648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2055.335611] Lustre: server umount lustre-MDT0001 complete [ 2059.256645] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2064.316539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2073.065611] Lustre: 63582:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441512/real 1788441512] req@000000005e7b30e5 x1875316737876224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788441528 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2073.113768] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2073.138448] Lustre: Skipped 1 previous similar message [ 2073.973230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2075.167162] Lustre: 63579:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441515/real 1788441515] req@00000000af37b99a x1875316737876352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788441531 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2077.279147] Lustre: 63579:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441517/real 1788441517] req@00000000c77117d8 x1875316737876480/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788441533 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2079.335987] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x5de9dfae6974501f [ 2079.349834] Lustre: MGC192.168.203.139@tcp: Connection restored to 0@lo (at 0@lo) [ 2079.773525] LustreError: 66507:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2079.792243] LustreError: 66507:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2079.863568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2079.970813] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2080.870095] Lustre: 63581:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441520/real 1788441520] req@0000000084792fa4 x1875316737876736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788441536 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2081.891849] LustreError: 66508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2083.941405] LustreError: 66507:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2083.983869] LustreError: 66507:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2084.851926] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2085.857850] Lustre: 63582:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441525/real 1788441525] req@000000005393cb5e x1875316737876864/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788441541 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2089.442562] LustreError: 68426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2089.478343] LustreError: 68426:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2093.747540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2094.152906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2099.030509] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2099.177093] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2101.229572] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2106.907142] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2112.932354] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2119.659558] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2119.672513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2119.693800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2119.704475] Lustre: Skipped 2 previous similar messages [ 2119.827402] Lustre: server umount lustre-MDT0000 complete [ 2121.696886] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2121.707261] LustreError: 66508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2121.722181] Lustre: Skipped 1 previous similar message [ 2121.756097] LustreError: 66508:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2123.314813] LustreError: 66513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441579 with bad export cookie 6767185855287414815 [ 2123.326674] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2123.327674] LustreError: 66513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2123.859772] Lustre: server umount lustre-MDT0001 complete [ 2130.078693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2139.068080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2142.177908] Lustre: 63579:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441582/real 1788441582] req@00000000dd362ef5 x1875316737903616/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788441598 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2142.223368] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2148.946376] LustreError: 66507:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2149.031413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2149.103682] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2154.199354] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2161.657835] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2163.924232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2169.286829] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2169.330330] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2177.867577] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2183.909899] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2194.914026] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2194.916695] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2194.925248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2194.941343] Lustre: Skipped 1 previous similar message [ 2199.976813] Lustre: server umount lustre-MDT0000 complete [ 2200.035076] LustreError: 66508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2200.051855] LustreError: 66508:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 2204.532889] LustreError: 64648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441660 with bad export cookie 6767185855287416194 [ 2204.555159] LustreError: 64648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2204.558210] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2205.119767] Lustre: server umount lustre-MDT0001 complete [ 2212.560490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2215.522132] Lustre: DEBUG MARKER: files compared the same [ 2227.551664] Lustre: server umount lustre-OST0000 complete [ 2242.268786] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 2244.555170] Key type lgssc unregistered [ 2244.875920] LNet: 74386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2244.881197] LNetError: 74386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2244.913145] LNet: Removed LNI 192.168.203.139@tcp [ 2245.789661] Key type .llcrypt unregistered [ 2245.793368] Key type ._llcrypt unregistered [ 2272.809934] Key type ._llcrypt registered [ 2272.813119] Key type .llcrypt registered [ 2272.973479] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2274.366506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2274.468837] alg: No test for adler32 (adler32-zlib) [ 2275.794205] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2276.142460] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 2277.855227] Key type lgssc registered [ 2279.193188] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2287.407134] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2287.448427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2289.188042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2294.935746] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug +malloc all [ 2299.359680] LustreError: 75798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2299.385336] LustreError: 75798:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2304.486927] LustreError: 75799:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2305.203070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2305.749897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2311.073902] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug +malloc all [ 2319.680974] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2325.789851] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2332.919291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2333.212851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2334.253924] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2339.848996] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug +malloc all [ 2349.259349] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2359.778746] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2359.783784] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2359.811802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2360.802727] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2360.809240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2365.398272] Lustre: server umount lustre-OST0000 complete [ 2371.041323] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2371.065135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2371.080539] Lustre: Skipped 2 previous similar messages [ 2375.677388] Lustre: server umount lustre-MDT0000 complete [ 2376.161491] LustreError: 75799:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2380.235251] LustreError: 75778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441836 with bad export cookie 10968622017375230883 [ 2380.247708] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2380.253507] LustreError: 75778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2380.806700] Lustre: server umount lustre-MDT0001 complete [ 2391.446202] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 2395.823667] Key type lgssc unregistered [ 2396.403791] LNet: 79331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2396.409752] LNetError: 79331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2396.429514] LNet: Removed LNI 192.168.203.139@tcp [ 2397.610652] Key type .llcrypt unregistered [ 2397.613337] Key type ._llcrypt unregistered [ 2424.041468] Key type ._llcrypt registered [ 2424.043711] Key type .llcrypt registered [ 2424.149912] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2425.148612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2425.193501] alg: No test for adler32 (adler32-zlib) [ 2426.294951] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2426.573235] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 2428.226205] Key type lgssc registered [ 2429.483507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2438.573865] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2438.609534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2454.239338] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2459.359600] LustreError: 80720:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.139@tcp: failed processing log, type 4: rc = -110 [ 2486.048176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2492.987522] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2501.149164] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2509.101292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2509.215522] Lustre: *** cfs_fail_loc=706, val=0*** [ 2509.744291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2509.849305] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2514.913653] LustreError: 80745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2514.934332] LustreError: 80745:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2514.967858] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2520.033227] LustreError: 80746:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2521.061119] LustreError: 81806:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2523.826590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2524.241856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2529.125932] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2536.528184] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2542.431984] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2544.616986] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2544.622636] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2544.628711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2544.656152] Lustre: Skipped 1 previous similar message [ 2549.737881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2549.752271] Lustre: Skipped 1 previous similar message [ 2550.776394] Lustre: server umount lustre-OST0000 complete [ 2554.850215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2554.856093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2554.856841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2554.868570] Lustre: Skipped 1 previous similar message [ 2559.969195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2559.976399] Lustre: Skipped 2 previous similar messages [ 2565.089731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2565.100707] Lustre: Skipped 1 previous similar message [ 2569.183739] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2569.398598] Lustre: server umount lustre-MDT0000 complete [ 2570.209029] LustreError: 81811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2570.239670] LustreError: 81811:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2573.688466] LustreError: 80726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442029 with bad export cookie 3411410590165786191 [ 2573.696359] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2573.699752] LustreError: 80726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2574.070137] Lustre: server umount lustre-MDT0001 complete [ 2585.502679] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 2588.470687] Key type lgssc unregistered [ 2588.899109] LNet: 84336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2588.916530] LNetError: 84336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2589.989737] LNet: Removed LNI 192.168.203.139@tcp [ 2591.015078] Key type .llcrypt unregistered [ 2591.017444] Key type ._llcrypt unregistered [ 2621.681280] Key type ._llcrypt registered [ 2621.683788] Key type .llcrypt registered [ 2621.834779] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2623.496444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2623.542354] alg: No test for adler32 (adler32-zlib) [ 2624.825330] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2625.364498] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 2627.040323] Key type lgssc registered [ 2628.285696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2636.695732] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2636.720971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2642.052750] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2650.067911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2650.528645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2654.981814] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2655.713840] LustreError: 86436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2655.726701] LustreError: 86436:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2660.840045] LustreError: 86439:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2660.862384] LustreError: 86439:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2663.877634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2664.295163] LustreError: 86436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2664.313064] LustreError: 86436:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2664.329731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2669.545172] LustreError: 86435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2669.559441] LustreError: 86435:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2670.793927] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2674.660365] LustreError: 87182:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2674.691536] LustreError: 87182:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2679.399932] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2679.414758] Lustre: MGS: Client 2de26d73-a1b6-4cbd-a891-3e1964c91f05 (at 0@lo) reconnecting [ 2679.433682] LustreError: 86450:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2679.774298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2679.890108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2684.892977] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2700.262046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2700.265601] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2700.282136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2700.295321] Lustre: Skipped 1 previous similar message [ 2705.405818] LustreError: 87164:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2705.429727] LustreError: 87164:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2705.511742] Lustre: server umount lustre-OST0000 complete [ 2710.503734] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2710.525762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2710.529651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2710.554338] Lustre: Skipped 2 previous similar messages [ 2710.574039] Lustre: Skipped 1 previous similar message [ 2715.617167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2715.630280] Lustre: Skipped 1 previous similar message [ 2715.774157] Lustre: server umount lustre-MDT0000 complete [ 2720.063841] Lustre: server umount lustre-MDT0001 complete [ 2724.494797] Lustre: server umount MGS complete [ 2736.975559] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:29:51 (1788442191) [ 2768.293118] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 2780.350147] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2819.523251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2824.664544] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2834.868255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2835.278857] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2835.339611] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2835.401388] Lustre: lustre-MDT0001: new disk, initializing [ 2835.463912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2835.496420] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2835.503759] Lustre: 93626:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2835.518451] LustreError: 93626:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2840.610692] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2851.581726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2851.920627] Lustre: lustre-OST0000: new disk, initializing [ 2851.930240] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2851.940841] Lustre: 94392:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2852.015995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2858.189143] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2859.040680] Lustre: 94722:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000f5b84e16 x1875317443567616/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788442330 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2868.925226] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2877.442120] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2877.450192] Lustre: MGS: Client cf08ece0-caf8-44d4-ba40-89d08778c143 (at 0@lo) reconnecting [ 2877.497534] Lustre: 92892:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2877.525256] LustreError: 93611:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2877.559134] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2877.670490] Lustre: lustre-MDT0000: new disk, initializing [ 2877.954830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2878.007441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2878.027603] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2879.074813] Lustre: 95461:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000035a98aef x1875317443574784/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788442351 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2883.395200] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2884.259167] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2888.176139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2888.187596] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2888.347407] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2914.272091] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2914.273870] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2914.282839] LustreError: Skipped 1 previous similar message [ 2914.283568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2914.314533] Lustre: Skipped 1 previous similar message [ 2919.547763] Lustre: server umount lustre-OST0000 complete [ 2928.108485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2928.109855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2928.111950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2928.111967] Lustre: Skipped 3 previous similar messages [ 2928.123622] Lustre: Skipped 1 previous similar message [ 2929.468169] Lustre: server umount lustre-MDT0000 complete [ 2933.217720] LustreError: 93596:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2933.245408] LustreError: 93596:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2939.756926] Lustre: server umount lustre-MDT0001 complete [ 2943.706618] Lustre: server umount MGS complete [ 2955.270671] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:33:29 (1788442409) [ 2975.259372] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 2978.381420] Key type lgssc unregistered [ 2978.775943] LNet: 97967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2978.793035] LNetError: 97967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2978.814606] LNet: Removed LNI 192.168.203.139@tcp [ 2979.765762] Key type .llcrypt unregistered [ 2979.767613] Key type ._llcrypt unregistered [ 2998.216878] Key type ._llcrypt registered [ 2998.221427] Key type .llcrypt registered [ 2998.321579] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2999.476340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2999.564317] alg: No test for adler32 (adler32-zlib) [ 3000.756756] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3001.078807] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 3002.839164] Key type lgssc registered [ 3004.065926] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3013.433508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3013.476664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3015.121802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3020.401325] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3025.377647] LustreError: 99184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3025.408679] LustreError: 99184:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3030.398371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3030.504220] LustreError: 99203:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3030.950943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3035.923974] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3043.797415] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3049.792460] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3056.987530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3057.389145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3060.475174] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3064.520438] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3074.122373] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3086.306303] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3086.317582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3086.337857] Lustre: Skipped 1 previous similar message [ 3086.350722] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3090.314743] Lustre: server umount lustre-OST0000 complete [ 3096.546472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3096.549891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3096.584078] Lustre: Skipped 1 previous similar message [ 3096.586080] Lustre: Skipped 1 previous similar message [ 3100.994884] Lustre: server umount lustre-MDT0000 complete [ 3101.664893] LustreError: 99185:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3101.689836] LustreError: 99185:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3105.237070] LustreError: 99166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442561 with bad export cookie 12967261138320555191 [ 3105.246987] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3105.249724] LustreError: 99166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3105.681993] Lustre: server umount lustre-MDT0001 complete [ 3117.653219] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 3120.530077] Key type lgssc unregistered [ 3120.825647] LNet: 102719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3120.837801] LNetError: 102719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3121.897471] LNet: Removed LNI 192.168.203.139@tcp [ 3122.757725] Key type .llcrypt unregistered [ 3122.761451] Key type ._llcrypt unregistered [ 3138.947262] Key type ._llcrypt registered [ 3138.951028] Key type .llcrypt registered [ 3139.081523] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 3140.837406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3140.907256] alg: No test for adler32 (adler32-zlib) [ 3141.932207] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3142.170220] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 3143.847304] Key type lgssc registered [ 3145.118564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3152.078527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3152.092442] LustreError: 103796:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3152.120089] LustreError: 103803:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3152.127051] LustreError: 103796:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4986 [ 3152.137055] LustreError: 103796:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3152.148885] LustreError: 103796:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3152.158854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3153.980858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3163.108608] LustreError: 103840:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3163.144656] LustreError: 103840:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3163.277838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3163.530684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3169.322731] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3174.954567] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing lsmod [ 3178.047170] LustreError: 104999:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3178.231780] LustreError: 105009:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3178.243716] LustreError: 104999:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4808 [ 3178.258605] LustreError: 104999:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3178.268474] LustreError: 104999:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3178.280772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3178.911402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3179.963360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3184.099791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3184.104743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3184.114044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3185.120967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3185.127117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3185.145700] Lustre: Skipped 1 previous similar message [ 3185.148763] Lustre: Skipped 1 previous similar message [ 3188.259541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3188.272231] Lustre: Skipped 3 previous similar messages [ 3193.317586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3195.871177] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3196.078225] Lustre: server umount lustre-MDT0000 complete [ 3198.437517] LustreError: 103835:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3198.467597] LustreError: 103835:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3200.134866] LustreError: 103821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442656 with bad export cookie 10990295533228618343 [ 3200.142751] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3200.148749] LustreError: 103821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3201.567576] LustreError: 104271:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000075387893 x1875317985663232/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3201.746640] Lustre: server umount lustre-MDT0001 complete [ 3216.104521] Lustre: server umount lustre-OST0000 complete [ 3227.432162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3228.215283] LustreError: 106076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3228.292580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3233.354728] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3233.760962] LustreError: 106077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3238.880935] LustreError: 106076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3242.662406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3243.254465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3248.044123] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3256.367552] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3262.758939] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3269.035628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3269.396878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3273.523957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3276.578847] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3284.350204] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3296.755604] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3296.769086] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3296.793479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3296.806039] Lustre: Skipped 2 previous similar messages [ 3297.273962] Lustre: server umount lustre-OST0000 complete [ 3304.425090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3304.455456] Lustre: Skipped 1 previous similar message [ 3307.028715] Lustre: server umount lustre-MDT0000 complete [ 3309.541645] LustreError: 106071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3309.552437] LustreError: 106071:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3310.821639] LustreError: 106058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442766 with bad export cookie 10990295533228619267 [ 3310.833542] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3310.833839] LustreError: 106058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3320.521831] Lustre: server umount lustre-MDT0001 complete [ 3331.231859] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 3333.766648] Key type lgssc unregistered [ 3334.077754] LNet: 109609:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3334.098039] LNetError: 109609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3334.128197] LNet: Removed LNI 192.168.203.139@tcp [ 3335.080610] Key type .llcrypt unregistered [ 3335.085794] Key type ._llcrypt unregistered [ 3361.005857] Key type ._llcrypt registered [ 3361.009984] Key type .llcrypt registered [ 3361.160933] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 3361.896679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3361.962664] alg: No test for adler32 (adler32-zlib) [ 3363.027929] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3363.262373] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 3364.983227] Key type lgssc registered [ 3365.915295] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3374.738913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3374.768547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3376.246031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3381.050239] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3386.856492] LustreError: 111025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3386.899482] LustreError: 111025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3390.472081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3390.933274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3395.993732] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3403.792061] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3409.650714] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3419.009696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3419.414351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3421.562232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3427.004994] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3435.173627] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3441.342328] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3444.943193] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3451.363758] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3451.366577] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3451.383991] Lustre: Skipped 1 previous similar message [ 3451.387512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3451.399557] Lustre: Skipped 1 previous similar message [ 3455.956557] Lustre: server umount lustre-OST0000 complete [ 3462.627212] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3462.643966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3462.665553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3466.722454] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3466.722813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3466.746040] Lustre: Skipped 1 previous similar message [ 3468.533395] Lustre: server umount lustre-MDT0000 complete [ 3471.844055] LustreError: 111022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3471.876449] LustreError: 111022:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3472.882409] LustreError: 111026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3472.930265] LustreError: 111026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3474.073862] LustreError: 113556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442930 with bad export cookie 14189493354090660402 [ 3474.078424] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3474.103907] LustreError: 113556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3474.539864] Lustre: server umount lustre-MDT0001 complete [ 3486.177943] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 3489.155254] Key type lgssc unregistered [ 3489.598673] LNet: 114670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3489.617888] LNetError: 114670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3489.655296] LNet: Removed LNI 192.168.203.139@tcp [ 3490.719903] Key type .llcrypt unregistered [ 3490.727141] Key type ._llcrypt unregistered [ 3508.599772] Key type ._llcrypt registered [ 3508.603478] Key type .llcrypt registered [ 3508.693860] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 3509.719598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3509.773569] alg: No test for adler32 (adler32-zlib) [ 3510.774489] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3511.092856] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 3512.847381] Key type lgssc registered [ 3514.232679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3523.496744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3523.565617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3525.125488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3530.731756] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3535.329709] LustreError: 115888:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3535.360567] LustreError: 115888:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3539.229174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3539.827881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3544.639618] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3552.053223] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3558.888734] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3566.994380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3567.437303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3570.586127] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3573.596894] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3582.383953] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3600.352383] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3600.366463] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3600.379735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3600.379858] Lustre: Skipped 1 previous similar message [ 3600.422617] Lustre: Skipped 1 previous similar message [ 3603.115257] Lustre: server umount lustre-OST0000 complete [ 3610.603179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3610.605299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3610.619932] Lustre: Skipped 1 previous similar message [ 3610.637189] Lustre: Skipped 1 previous similar message [ 3612.957410] Lustre: server umount lustre-MDT0000 complete [ 3615.712930] LustreError: 115889:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3615.725911] LustreError: 115889:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3616.476264] LustreError: 115869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443072 with bad export cookie 12204038084524407214 [ 3616.477053] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3616.505834] LustreError: 115869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3616.719890] Lustre: server umount lustre-MDT0001 complete [ 3629.083970] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 3632.300805] Key type lgssc unregistered [ 3632.742055] LNet: 119441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3632.752742] LNetError: 119441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3633.770544] LNet: Removed LNI 192.168.203.139@tcp [ 3634.787060] Key type .llcrypt unregistered [ 3634.788661] Key type ._llcrypt unregistered [ 3659.954136] Key type ._llcrypt registered [ 3659.957304] Key type .llcrypt registered [ 3660.106631] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 3661.536655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3661.574718] alg: No test for adler32 (adler32-zlib) [ 3662.743375] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3663.056243] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 3664.727453] Key type lgssc registered [ 3665.896748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3673.657985] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3673.698195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3675.242963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3680.058306] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3685.345222] LustreError: 120853:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3685.371829] LustreError: 120853:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3689.249990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3689.734747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3694.178775] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3701.713970] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3709.504414] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3718.648986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3719.274836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3720.320686] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3726.998715] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3735.772925] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3744.453600] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3748.438170] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3748.448484] Lustre: Skipped 2 previous similar messages [ 3753.075543] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3753.083090] Lustre: Skipped 2 previous similar messages [ 3753.509403] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3753.543256] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3759.491655] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.39@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3759.498454] Lustre: Skipped 2 previous similar messages [ 3761.858604] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.39@tcp 192.168.0.[2,10]@tcp [ 3768.359112] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.39/24@tcp in log lustre-MDT0000 [ 3768.373920] Lustre: Skipped 2 previous similar messages [ 3774.547836] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.39/24@tcp [ 3774.565107] Lustre: Skipped 1 previous similar message [ 3781.092460] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3781.097511] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3781.108232] Lustre: Skipped 1 previous similar message [ 3781.110926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3781.132603] Lustre: Skipped 1 previous similar message [ 3785.275722] Lustre: server umount lustre-OST0000 complete [ 3791.335590] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3791.338818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3791.345251] Lustre: Skipped 1 previous similar message [ 3791.354104] Lustre: Skipped 1 previous similar message [ 3795.417725] Lustre: server umount lustre-MDT0000 complete [ 3796.456858] LustreError: 120848:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3796.478930] LustreError: 120848:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3799.096093] LustreError: 120832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443255 with bad export cookie 18201103601841186391 [ 3799.097161] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3799.106209] LustreError: 120832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3799.336958] Lustre: server umount lustre-MDT0001 complete [ 3809.842276] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 3812.583306] Key type lgssc unregistered [ 3812.863293] LNet: 125591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3812.872410] LNetError: 125591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3812.891186] LNet: Removed LNI 192.168.203.139@tcp [ 3813.707099] Key type .llcrypt unregistered [ 3813.715978] Key type ._llcrypt unregistered [ 3840.632810] Key type ._llcrypt registered [ 3840.640395] Key type .llcrypt registered [ 3840.768612] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 3841.831032] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3841.850235] alg: No test for adler32 (adler32-zlib) [ 3842.914899] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3843.127837] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 3844.847493] Key type lgssc registered [ 3846.165121] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3853.794407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3853.832282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3855.378295] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3855.412373] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.39/24@tcp [ 3855.421121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3860.300528] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3865.579915] LustreError: 127150:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3865.601599] LustreError: 127150:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3869.647971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3869.892167] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3869.919088] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.39/24@tcp [ 3869.962269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3875.105446] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3883.008271] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3889.326818] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3896.114605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3896.469678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3900.622325] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3902.997957] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3911.238256] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3924.970675] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3924.978813] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3924.995914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3925.492978] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3925.508767] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3925.526286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3925.536772] Lustre: Skipped 1 previous similar message [ 3930.440984] Lustre: server umount lustre-OST0000 complete [ 3935.713107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3935.717710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3935.737000] Lustre: Skipped 1 previous similar message [ 3935.745874] Lustre: Skipped 1 previous similar message [ 3940.604363] Lustre: server umount lustre-MDT0000 complete [ 3940.834905] LustreError: 127150:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3940.851097] LustreError: 127150:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3944.772647] LustreError: 127130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443400 with bad export cookie 6096635763097389630 [ 3944.781801] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3944.785676] LustreError: 127130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3945.210233] Lustre: server umount lustre-MDT0001 complete [ 3956.569586] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 3959.406822] Key type lgssc unregistered [ 3959.715896] LNet: 130739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3959.724726] LNetError: 130739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3959.750732] LNet: Removed LNI 192.168.203.139@tcp [ 3960.695447] Key type .llcrypt unregistered [ 3960.705211] Key type ._llcrypt unregistered [ 3985.574553] Key type ._llcrypt registered [ 3985.578436] Key type .llcrypt registered [ 3985.727199] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 3999.612176] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 4000.444598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4000.475812] alg: No test for adler32 (adler32-zlib) [ 4001.473155] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4001.686082] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 4003.331152] Key type lgssc registered [ 4004.516042] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4038.535318] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4038.547960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4039.695641] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4039.712868] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4039.768939] Lustre: lustre-MDT0000: new disk, initializing [ 4039.828261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4039.842971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4044.443909] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4054.732274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4054.847753] Lustre: 134134:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4054.904496] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4054.914862] Lustre: Skipped 1 previous similar message [ 4055.008653] Lustre: lustre-MDT0001: new disk, initializing [ 4055.112720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4055.150774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4055.158208] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4059.407796] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4069.207029] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4074.962728] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4083.360302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4083.885470] Lustre: lustre-OST0000: new disk, initializing [ 4083.888988] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4083.895045] Lustre: 136101:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4083.974298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4091.428170] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4093.071604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4093.098709] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4093.347115] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4102.789978] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4108.048108] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4108.317395] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4111.923941] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4112.055756] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4113.893611] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4113.895507] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4113.895962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4113.908078] Lustre: Skipped 1 previous similar message [ 4113.910676] LustreError: Skipped 1 previous similar message [ 4119.012523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4119.029099] Lustre: Skipped 2 previous similar messages [ 4124.134758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4124.150951] Lustre: Skipped 1 previous similar message [ 4128.223157] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4128.688090] Lustre: server umount lustre-OST0000 complete [ 4134.368684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4134.394641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4134.400113] Lustre: Skipped 1 previous similar message [ 4139.496576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4139.512951] Lustre: Skipped 1 previous similar message [ 4140.580379] Lustre: server umount lustre-MDT0000 complete [ 4143.833514] LustreError: 134126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443599 with bad export cookie 7816377085150683991 [ 4143.837047] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4143.845689] LustreError: 134126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4144.224062] Lustre: server umount lustre-MDT0001 complete [ 4153.491687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4153.998254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4157.233090] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4159.457440] LustreError: 138322:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4159.490066] LustreError: 138322:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4163.554941] LustreError: 138321:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4166.558901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4167.125829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4172.134279] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4180.845326] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4187.215916] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4193.836867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4194.279094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4201.585624] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4210.010570] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4216.347596] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4216.620772] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4222.181923] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4222.370500] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4233.875641] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4234.134874] Lustre: lustre-OST0001: new disk, initializing [ 4234.154576] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4234.172754] Lustre: 141696:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4234.336659] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4241.737862] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4242.182407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4242.190102] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4242.519816] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4251.418958] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4251.827605] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4257.579818] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4257.871469] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4264.893543] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c40476be800.ost_server_uuid 50 [ 4266.382197] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8c40476be800.ost_server_uuid in IDLE state after 0 sec [ 4278.274743] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4278.291732] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4278.324862] Lustre: Skipped 1 previous similar message [ 4278.335204] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4278.355714] Lustre: Skipped 1 previous similar message [ 4279.561357] Lustre: server umount lustre-OST0001 complete [ 4283.360073] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4283.366943] LustreError: Skipped 1 previous similar message [ 4283.367333] LustreError: 140143:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4283.369972] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4283.369981] Lustre: Skipped 1 previous similar message [ 4283.423140] LustreError: 140143:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4288.506180] LustreError: 140141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4288.522309] LustreError: 140141:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4289.689431] Lustre: server umount lustre-OST0000 complete [ 4293.600234] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4293.609253] Lustre: Skipped 2 previous similar messages [ 4298.720645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4298.727488] Lustre: Skipped 6 previous similar messages [ 4299.775066] Lustre: server umount lustre-MDT0000 complete [ 4303.841770] LustreError: 138322:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4303.870801] LustreError: 138322:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4304.562857] LustreError: 138303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443760 with bad export cookie 7816377085150684873 [ 4304.563825] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4304.573405] LustreError: 138303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4305.095515] Lustre: server umount lustre-MDT0001 complete [ 4329.077225] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 4332.005497] Key type lgssc unregistered [ 4332.238530] LNet: 144697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4332.245672] LNetError: 144697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4332.266265] LNet: Removed LNI 192.168.203.139@tcp [ 4333.005202] Key type .llcrypt unregistered [ 4333.007707] Key type ._llcrypt unregistered [ 4375.040559] Key type ._llcrypt registered [ 4375.042162] Key type .llcrypt registered [ 4375.179814] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 4392.226522] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 4393.283242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4393.423484] alg: No test for adler32 (adler32-zlib) [ 4394.623995] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4394.941797] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 4396.631223] Key type lgssc registered [ 4397.704280] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4435.951627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4435.970571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4437.324521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4437.378273] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4437.447988] Lustre: lustre-MDT0000: new disk, initializing [ 4437.519834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4437.545668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4443.202705] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4454.727764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4454.857733] Lustre: 148849:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4454.906374] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4454.910890] Lustre: Skipped 1 previous similar message [ 4454.968460] Lustre: lustre-MDT0001: new disk, initializing [ 4455.042846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4455.091654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4455.106178] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4460.043132] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4470.276689] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4475.682362] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4483.049970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4483.212894] Lustre: lustre-OST0000: new disk, initializing [ 4483.216134] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4483.221692] Lustre: 150819:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4483.307064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4489.598671] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4490.347146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4490.364317] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4490.547769] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4501.626913] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4506.960322] Lustre: Failing over lustre-OST0000 [ 4507.397432] Lustre: server umount lustre-OST0000 complete [ 4526.052723] Lustre: 147070:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443965/real 1788443965] req@00000000ba5efabd x1875319298493312/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788443981 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4526.096859] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4527.144375] Lustre: 147072:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443967/real 1788443967] req@0000000030240a8e x1875319298493440/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788443983 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4527.180400] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4527.863748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4528.307435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4528.355113] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4532.319429] Lustre: 147071:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443972/real 1788443972] req@00000000b8fb6f5c x1875319298495360/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788443988 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4532.385276] Lustre: 147071:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4535.700988] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4535.900761] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4537.442336] Lustre: 147072:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443977/real 1788443977] req@00000000999b6fab x1875319298496640/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788443993 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4537.465070] Lustre: 147072:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4537.491538] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4537.494368] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4539.895248] Lustre: Failing over lustre-MDT0000 [ 4540.313754] Lustre: server umount lustre-MDT0000 complete [ 4541.476617] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4541.490465] Lustre: Skipped 2 previous similar messages [ 4542.495204] Lustre: 147072:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443982/real 1788443982] req@00000000c5d529bb x1875319298497664/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788443998 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4542.543369] Lustre: 147072:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4546.137715] LustreError: 149980:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4546.178193] LustreError: 149980:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4551.255636] LustreError: 148855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4551.267908] LustreError: 148855:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4556.376296] LustreError: 149980:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4556.428073] LustreError: 149980:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4557.791496] Lustre: 147070:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443997/real 1788443997] req@000000000047dbc5 x1875319298508672/t0(0) o400->MGC192.168.203.139@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788444013 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4557.852508] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4561.495686] LustreError: 148854:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4561.540104] LustreError: 148854:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4564.284091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4566.624275] LustreError: 148855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4566.643946] LustreError: 148855:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4567.074109] LustreError: 147069:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000024e395f9 x1875319298514176/t0(0) o250->MGC192.168.203.139@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4567.347530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4567.398289] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4571.736430] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4572.654311] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4572.669163] Lustre: Skipped 1 previous similar message [ 4572.688583] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4572.732410] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4573.153440] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4588.010709] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4588.020583] Lustre: Skipped 1 previous similar message [ 4588.024373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4588.051965] Lustre: Skipped 1 previous similar message [ 4593.133070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4593.148880] Lustre: Skipped 1 previous similar message [ 4598.245478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4598.249144] Lustre: Skipped 1 previous similar message [ 4601.311112] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4601.539843] Lustre: server umount lustre-OST0000 complete [ 4608.484542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4608.492700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4608.508270] Lustre: Skipped 1 previous similar message [ 4612.031522] Lustre: server umount lustre-MDT0000 complete [ 4613.605926] LustreError: 148860:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4613.626863] LustreError: 148860:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4616.503758] LustreError: 149644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444072 with bad export cookie 153390297852902711 [ 4616.505048] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4616.514207] LustreError: 149644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4617.066379] Lustre: server umount lustre-MDT0001 complete [ 4629.673569] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 4633.446735] Key type lgssc unregistered [ 4633.844707] LNet: 154487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4633.856754] LNetError: 154487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4633.901256] LNet: Removed LNI 192.168.203.139@tcp [ 4635.320627] Key type .llcrypt unregistered [ 4635.322632] Key type ._llcrypt unregistered [ 4682.497484] Key type ._llcrypt registered [ 4682.500621] Key type .llcrypt registered [ 4682.623448] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 4683.766922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4683.959635] alg: No test for adler32 (adler32-zlib) [ 4685.145534] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4685.437656] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 4687.159261] Key type lgssc registered [ 4688.238157] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4696.738803] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4696.754677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4698.214070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4702.511711] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4708.839894] LustreError: 156009:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4708.865456] LustreError: 156009:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4713.872531] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4713.952903] LustreError: 156008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4717.989196] LustreError: 156037:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4718.014666] LustreError: 156037:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4718.025307] LustreError: 156037:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4718.419519] Lustre: server umount lustre-MDT0000 complete [ 4737.398126] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 4749.834416] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 4790.324855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4790.608734] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4790.645427] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4790.760284] Lustre: lustre-MDT0000: new disk, initializing [ 4790.856492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4790.881343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4795.348564] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4805.366878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4805.515802] Lustre: 159343:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4805.585031] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4805.589118] Lustre: Skipped 1 previous similar message [ 4805.654600] Lustre: lustre-MDT0001: new disk, initializing [ 4805.746491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4805.796509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4805.812088] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4810.608899] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4820.236693] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4826.282558] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4833.511354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4833.788664] Lustre: lustre-OST0000: new disk, initializing [ 4833.794904] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4833.802767] Lustre: 161310:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4833.903073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4839.525869] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4840.023546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4840.029638] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4840.259406] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4850.707757] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4856.632432] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4859.306532] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 10:05:13 (1788444313) [ 5147.620578] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5147.626577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5147.640621] Lustre: Skipped 2 previous similar messages [ 5147.648523] Lustre: Skipped 2 previous similar messages [ 5152.750683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5152.765419] Lustre: Skipped 2 previous similar messages [ 5153.113566] Lustre: server umount lustre-MDT0000 complete [ 5157.297531] LustreError: 159335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444613 with bad export cookie 11284450162583871949 [ 5157.307850] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5157.310485] LustreError: 159335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5157.855510] LustreError: 161318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5157.855597] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5157.897504] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5162.984344] LustreError: 159350:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5162.984625] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5163.678983] Lustre: server umount lustre-MDT0001 complete [ 5174.428550] Lustre: server umount lustre-OST0000 complete [ 5185.505721] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 5196.886544] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 5237.193941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5237.474104] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5237.517175] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5237.602672] Lustre: lustre-MDT0000: new disk, initializing [ 5237.705240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5237.725967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5242.384970] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5253.785355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5253.977784] Lustre: 165980:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5254.064823] LustreError: 165990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5254.094390] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5254.104714] Lustre: Skipped 1 previous similar message [ 5254.175930] Lustre: lustre-MDT0001: new disk, initializing [ 5254.275234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5254.324757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5254.346567] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5258.917542] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5268.762656] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5276.279398] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5285.216324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5285.627253] Lustre: lustre-OST0000: new disk, initializing [ 5285.632123] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5285.647699] Lustre: 167948:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5285.796028] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5294.215320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5294.227060] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5294.306447] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5294.479969] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5306.387082] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5311.873584] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5312.053902] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5317.211955] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5317.460930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5320.160099] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5320.163056] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5320.164041] LustreError: Skipped 1 previous similar message [ 5320.164946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5320.204558] Lustre: Skipped 1 previous similar message [ 5325.658865] Lustre: server umount lustre-OST0000 complete [ 5330.407283] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5330.410843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5330.426786] Lustre: Skipped 1 previous similar message [ 5330.450497] Lustre: Skipped 4 previous similar messages [ 5336.158467] Lustre: server umount lustre-MDT0000 complete [ 5340.085214] LustreError: 165972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444796 with bad export cookie 11284450162584030786 [ 5340.090386] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5340.108272] LustreError: 165972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5340.432688] Lustre: server umount lustre-MDT0001 complete [ 5350.614271] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:13:24 (1788444804) [ 5369.402252] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 5379.892420] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 5417.555368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5417.921400] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5417.961542] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5418.029450] Lustre: lustre-MDT0000: new disk, initializing [ 5418.146912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5418.179825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5423.044985] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5435.228206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5435.541991] Lustre: 172710:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5435.882576] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5440.878197] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5451.099893] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5457.637355] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5467.078278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5467.390808] Lustre: lustre-OST0000: new disk, initializing [ 5467.394674] Lustre: Skipped 1 previous similar message [ 5467.400918] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5467.412079] Lustre: Skipped 2 previous similar messages [ 5467.417411] Lustre: 174686:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5475.199850] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5477.012741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5477.037092] Lustre: Skipped 1 previous similar message [ 5477.049881] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5477.227697] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5487.321542] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5497.833282] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5497.840873] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5497.847868] LustreError: Skipped 1 previous similar message [ 5497.854285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5497.873790] Lustre: Skipped 1 previous similar message [ 5497.897886] Lustre: Skipped 3 previous similar messages [ 5503.749748] Lustre: server umount lustre-OST0000 complete [ 5512.679436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5512.687173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5514.783090] Lustre: server umount lustre-MDT0000 complete [ 5518.308412] LustreError: 172720:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5518.333512] LustreError: 172720:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5519.533049] LustreError: 172702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444975 with bad export cookie 11284450162584031668 [ 5519.536281] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5519.551498] LustreError: 172702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5520.079811] Lustre: server umount lustre-MDT0001 complete [ 5530.696817] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:16:24 (1788444984) [ 5546.967545] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 5558.095212] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 5600.991125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5601.289319] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5601.343903] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5601.414950] Lustre: lustre-MDT0000: new disk, initializing [ 5601.481700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5601.497394] Lustre: Skipped 2 previous similar messages [ 5601.532540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5605.211520] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5616.216339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5616.304482] Lustre: 179012:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5616.663696] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5621.763566] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5631.429554] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5636.993269] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5644.196062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5644.644551] Lustre: 180987:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5652.396751] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5653.640473] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5653.874070] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5664.486578] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5674.469883] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5674.482684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5674.486741] Lustre: Skipped 2 previous similar messages [ 5674.487744] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5674.507343] Lustre: Skipped 6 previous similar messages [ 5686.239202] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5686.496726] Lustre: server umount lustre-OST0000 complete [ 5693.410400] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5693.416871] LustreError: Skipped 1 previous similar message [ 5705.184321] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5705.495435] Lustre: server umount lustre-MDT0000 complete [ 5710.026546] LustreError: 179004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445165 with bad export cookie 11284450162584032753 [ 5710.034228] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5710.039398] LustreError: 179004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5710.322973] Lustre: server umount lustre-MDT0001 complete [ 5724.195927] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 5728.520892] Key type lgssc unregistered [ 5728.770296] LNet: 182847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5728.786147] LNetError: 182847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5729.829984] LNet: Removed LNI 192.168.203.139@tcp [ 5731.002819] Key type .llcrypt unregistered [ 5731.006251] Key type ._llcrypt unregistered [ 5756.493450] Key type ._llcrypt registered [ 5756.497815] Key type .llcrypt registered [ 5756.615084] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 5758.029572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5758.066474] alg: No test for adler32 (adler32-zlib) [ 5759.228274] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5759.550660] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 5761.295177] Key type lgssc registered [ 5762.717839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5771.858043] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5771.912688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5773.452116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5778.306347] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5784.037434] LustreError: 184271:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5784.065799] LustreError: 184271:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5787.253130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5787.690650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5792.797070] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5800.451834] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5806.444970] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5814.274317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5814.749911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5818.877533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5820.905967] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5828.607308] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5852.647299] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5852.650126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5852.666183] Lustre: Skipped 1 previous similar message [ 5857.303668] Lustre: server umount lustre-OST0000 complete [ 5862.886815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5862.894455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5862.900052] Lustre: Skipped 1 previous similar message [ 5862.914581] Lustre: Skipped 2 previous similar messages [ 5868.001579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5868.014523] Lustre: Skipped 1 previous similar message [ 5868.261026] Lustre: server umount lustre-MDT0000 complete [ 5872.098025] LustreError: 184271:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5872.120697] LustreError: 184271:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5872.473682] LustreError: 187074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445328 with bad export cookie 16165480584334719198 [ 5872.483722] LustreError: 187074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5872.485943] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5872.851878] Lustre: server umount lustre-MDT0001 complete [ 5885.197660] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 5888.761540] Key type lgssc unregistered [ 5889.180534] LNet: 187824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5889.187951] LNetError: 187824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5889.211890] LNet: Removed LNI 192.168.203.139@tcp [ 5890.219097] Key type .llcrypt unregistered [ 5890.224668] Key type ._llcrypt unregistered [ 5917.183281] Key type ._llcrypt registered [ 5917.185823] Key type .llcrypt registered [ 5917.337523] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 5918.822729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5918.932833] alg: No test for adler32 (adler32-zlib) [ 5920.010321] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5920.303357] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 5921.991229] Key type lgssc registered [ 5923.186819] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5931.269473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5931.298842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5932.736758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5936.964598] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5942.243359] LustreError: 189239:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5942.271856] LustreError: 189239:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5946.242284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5946.554513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5950.090787] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5958.300780] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5965.612970] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5974.255234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5974.600972] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5977.751496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5980.787953] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5988.738379] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5996.514037] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5996.514431] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5996.525168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5996.546533] Lustre: Skipped 1 previous similar message [ 5996.565942] Lustre: Skipped 1 previous similar message [ 6001.644148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6001.651151] Lustre: Skipped 2 previous similar messages [ 6001.977590] Lustre: server umount lustre-OST0000 complete [ 6009.766337] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8c4043124800.ost_server_uuid 50 [ 6011.823511] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6028.794811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6028.815703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6028.870593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6028.884943] Lustre: Skipped 1 previous similar message [ 6031.334335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6031.334468] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6033.896939] Lustre: server umount lustre-MDT0000 complete [ 6036.453753] LustreError: 189238:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6036.492256] LustreError: 189238:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6038.766434] LustreError: 189217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445494 with bad export cookie 17728056288737527234 [ 6038.770902] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6038.788666] LustreError: 189217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6039.072891] Lustre: server umount lustre-MDT0001 complete [ 6048.022224] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:25:02 (1788445502) [ 6055.034492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6055.432279] LustreError: 193154:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6055.543363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6060.360902] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6061.039337] LustreError: 193155:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6065.121024] LustreError: 193154:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6068.765149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6069.232317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6073.733731] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6081.713574] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6087.976761] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6095.750581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6096.085768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6100.216414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6102.965704] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6111.091264] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6119.780357] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6120.023242] Lustre: lustre-OST0001: new disk, initializing [ 6120.025805] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6120.031934] Lustre: 195997:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6120.129277] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6125.766414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6125.788389] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6125.875643] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6127.805311] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6142.399634] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6146.532940] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6146.540828] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6146.558274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6146.586981] Lustre: Skipped 1 previous similar message [ 6146.606985] Lustre: Skipped 2 previous similar messages [ 6156.773429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6156.780676] Lustre: Skipped 5 previous similar messages [ 6160.351143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6160.575626] Lustre: server umount lustre-OST0000 complete [ 6161.896948] LustreError: 194974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6161.928429] LustreError: 194974:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6167.475715] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6167.697590] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6172.860970] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6173.101172] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6179.427043] LustreError: 196243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6179.452367] LustreError: 196243:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6187.489751] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6187.496182] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6187.504138] LustreError: Skipped 1 previous similar message [ 6187.530458] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6187.535153] Lustre: Skipped 2 previous similar messages [ 6193.463257] Lustre: server umount lustre-OST0001 complete [ 6197.218269] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6197.225183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6197.243210] Lustre: Skipped 1 previous similar message [ 6203.540711] Lustre: server umount lustre-MDT0000 complete [ 6207.026939] LustreError: 195996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445662 with bad export cookie 17728056288737528298 [ 6207.034986] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6207.057614] LustreError: 195996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6207.390371] Lustre: server umount lustre-MDT0001 complete [ 6234.183884] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:28:08 (1788445688) [ 6241.788967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6242.204567] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6242.250775] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6242.462921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6248.173973] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6258.510946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6258.683604] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6258.775632] Lustre: 199440:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6263.724565] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6273.620048] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6279.687147] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6287.223647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6287.483288] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6294.406764] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6296.252345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6303.222626] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6310.767354] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6310.887673] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6311.142559] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6311.149973] Lustre: Skipped 2 previous similar messages [ 6317.719938] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6324.909824] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6331.042368] Lustre: lustre-OST0000: Not available for connect from 192.168.203.39@tcp (stopping) [ 6331.055792] Lustre: Skipped 6 previous similar messages [ 6331.878443] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6331.894743] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6331.922018] Lustre: Skipped 1 previous similar message [ 6334.409379] Lustre: server umount lustre-OST0000 complete [ 6336.089381] LustreError: 201289:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6336.115727] LustreError: 201289:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 6352.885772] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6352.904828] LustreError: Skipped 2 previous similar messages [ 6357.557358] Lustre: server umount lustre-OST0001 complete [ 6368.104669] Lustre: server umount lustre-MDT0000 complete [ 6372.066751] LustreError: 199434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445828 with bad export cookie 17728056288737529712 [ 6372.068387] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6372.077781] LustreError: 199434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6372.523587] Lustre: server umount lustre-MDT0001 complete [ 6397.863310] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:30:52 (1788445852) [ 6414.514324] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 6424.532876] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 6465.736066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6465.944402] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6465.985293] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6466.064791] Lustre: lustre-MDT0000: new disk, initializing [ 6466.136702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6466.175895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6471.172513] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6483.497393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6483.624878] Lustre: 207540:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6483.681695] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6483.685727] Lustre: Skipped 1 previous similar message [ 6483.757343] Lustre: lustre-MDT0001: new disk, initializing [ 6483.860913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6483.880892] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6489.203540] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6498.632901] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6504.453249] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6511.474753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6511.813546] Lustre: lustre-OST0000: new disk, initializing [ 6511.818966] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6511.839146] Lustre: 209507:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6517.823315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6517.841066] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6517.965931] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6518.186473] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6528.507710] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6533.452324] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6533.663315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6537.275661] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6537.419492] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6543.861538] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6543.879483] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6543.882826] Lustre: Skipped 5 previous similar messages [ 6543.887451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6543.913437] Lustre: Skipped 8 previous similar messages [ 6545.525188] Lustre: server umount lustre-OST0000 complete [ 6557.095806] Lustre: server umount lustre-MDT0000 complete [ 6559.206849] LustreError: 207550:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6559.233039] LustreError: 207550:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 6561.518245] LustreError: 207531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446017 with bad export cookie 17728056288737531273 [ 6561.539996] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6561.545888] LustreError: 207531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6562.070197] Lustre: server umount lustre-MDT0001 complete [ 6571.346922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6576.949640] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6586.188468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6591.111803] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6600.465909] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6606.554642] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6615.985789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6622.705291] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6634.460313] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6641.275262] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6641.645432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6647.560885] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6647.784199] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6652.395596] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6652.400368] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6652.412405] LustreError: Skipped 1 previous similar message [ 6652.470712] Lustre: Skipped 4 previous similar messages [ 6664.160492] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6664.399482] Lustre: server umount lustre-OST0000 complete [ 6671.363944] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6675.977966] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6706.106194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6713.529491] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6717.804787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6717.821418] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6717.824433] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6722.066326] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6724.026495] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6733.291403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6733.305638] Lustre: Skipped 8 previous similar messages [ 6735.767297] Lustre: server umount lustre-OST0000 complete [ 6755.296534] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6755.596982] Lustre: server umount lustre-MDT0000 complete [ 6758.882486] LustreError: 211727:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6758.914317] LustreError: 211727:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6760.349782] LustreError: 211706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446216 with bad export cookie 17728056288737532155 [ 6760.367797] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6760.371412] LustreError: 211706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6760.826578] Lustre: server umount lustre-MDT0001 complete [ 6770.790176] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:37:05 (1788446225) [ 6779.194226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6780.027529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6780.034196] Lustre: Skipped 6 previous similar messages [ 6785.182712] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6794.907928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6800.332460] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6808.734982] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6814.741357] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6822.033909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6828.971055] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6836.664892] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6841.606107] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6841.893570] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6846.364254] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6846.693663] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6853.553447] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6853.779478] Lustre: lustre-OST0001: new disk, initializing [ 6853.785186] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6853.800195] Lustre: 221065:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6860.124513] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6862.399984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6862.408930] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6862.491455] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6869.486253] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6874.352279] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6874.586273] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6878.901789] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6879.145843] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6883.299945] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6883.309046] LustreError: Skipped 2 previous similar messages [ 6883.311339] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6883.330315] Lustre: Skipped 5 previous similar messages [ 6887.263774] Lustre: server umount lustre-OST0001 complete [ 6894.198403] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6894.442484] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6898.846546] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6899.012558] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6928.781199] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6934.509769] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6934.555814] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6934.557585] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6934.600394] Lustre: Skipped 2 previous similar messages [ 6937.668897] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6946.099705] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6947.711364] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6955.481897] Lustre: server umount lustre-OST0001 complete [ 6987.231805] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6987.406757] Lustre: server umount lustre-OST0000 complete [ 6995.424628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6995.434500] Lustre: Skipped 21 previous similar messages [ 6998.000091] Lustre: server umount lustre-MDT0000 complete [ 7001.968176] LustreError: 217679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446457 with bad export cookie 17728056288737533121 [ 7001.976709] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7001.981232] LustreError: 217679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7008.371536] Lustre: server umount lustre-MDT0001 complete [ 7031.469269] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:41:26 (1788446486) [ 7037.312785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7037.565133] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7037.604517] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7042.621681] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7051.023940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7051.101802] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7051.120838] Lustre: 226315:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7055.462867] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7063.148953] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7068.743132] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7074.920789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7075.194913] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7082.095101] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7089.841607] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7098.663284] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7098.722590] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7105.515093] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7113.857804] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7115.488789] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7120.796766] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7121.113743] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7125.867187] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7126.162717] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7131.367461] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c40505d8000.ost_server_uuid 50 [ 7132.756860] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8c40505d8000.ost_server_uuid in IDLE state after 0 sec [ 7133.679362] Lustre: Permanently deactivating lustre-OST0001 [ 7133.688645] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7136.973038] Lustre: Permanently reactivating lustre-OST0001 [ 7141.858757] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7141.859473] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7141.868735] LustreError: Skipped 6 previous similar messages [ 7141.877249] Lustre: Skipped 6 previous similar messages [ 7147.549761] Lustre: server umount lustre-OST0001 complete [ 7152.096959] LustreError: 228160:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7152.119711] LustreError: 228160:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 41 previous similar messages [ 7158.057047] Lustre: server umount lustre-OST0000 complete [ 7176.159480] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7176.339530] Lustre: server umount lustre-MDT0000 complete [ 7179.966729] LustreError: 226307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446635 with bad export cookie 17728056288737534409 [ 7179.982253] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7179.987252] LustreError: 226307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7180.391503] Lustre: server umount lustre-MDT0001 complete [ 7201.905557] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:44:16 (1788446656) [ 7208.465840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7208.690369] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7208.718583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7208.725747] Lustre: Skipped 5 previous similar messages [ 7213.440718] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7221.903808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7221.999674] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7222.036413] Lustre: 233010:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7226.221176] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7234.285163] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7241.647713] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7250.053485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7250.252288] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7250.299179] Lustre: Permanently deactivating lustre-OST0000 [ 7250.317115] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7258.104689] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7258.633980] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7258.645753] LustreError: 235256:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0001: cannot cleanup orphans: rc = -5 [ 7266.056746] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7273.132248] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7273.184345] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7278.719195] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7279.055352] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7286.124716] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7287.899227] Lustre: Permanently reactivating lustre-OST0000 [ 7287.910718] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7287.915803] Lustre: Skipped 2 previous similar messages [ 7298.127103] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7298.128899] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 40 seconds. I think it's dead, and I am evicting it. exp 00000000e5d38d3d, cur 1788446754 deadline 1788446744 last 1788446714 [ 7298.138152] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7298.172565] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7298.237223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7307.481154] Lustre: server umount lustre-OST0001 complete [ 7317.241839] Lustre: server umount lustre-OST0000 complete [ 7335.391186] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7335.624674] Lustre: server umount lustre-MDT0000 complete [ 7338.803321] LustreError: 233001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446794 with bad export cookie 17728056288737536054 [ 7338.806304] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7338.821911] LustreError: 233001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7339.105444] Lustre: server umount lustre-MDT0001 complete [ 7349.411762] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7352.133938] Key type lgssc unregistered [ 7352.498974] LNet: 237893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7352.507354] LNetError: 237893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7352.529948] LNet: Removed LNI 192.168.203.139@tcp [ 7353.543668] Key type .llcrypt unregistered [ 7353.547349] Key type ._llcrypt unregistered [ 7374.821809] Key type ._llcrypt registered [ 7374.825247] Key type .llcrypt registered [ 7374.942730] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7376.021060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7376.045143] alg: No test for adler32 (adler32-zlib) [ 7377.158106] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7377.445820] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 7379.104312] Key type lgssc registered [ 7380.267912] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7390.870592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7390.890529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7392.482851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7397.521141] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7402.976890] LustreError: 239354:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7402.996870] LustreError: 239354:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7407.251968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7407.512494] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7407.566414] Lustre: 239342:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7407.583512] Lustre: Permanently deactivating lustre-MDT0001 [ 7407.609767] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7407.984883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7410.031177] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7410.043212] LustreError: 239384:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7410.054574] LustreError: 239384:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7410.066513] LustreError: 239384:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 1, failed: rc = -108 [ 7412.556542] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7420.020989] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7425.313392] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7432.507308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7432.684767] LustreError: 241197:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7432.840037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7437.928689] LustreError: 241197:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7437.953329] LustreError: 241197:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7437.969194] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7439.284932] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7443.456638] LustreError: 241199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7443.480129] LustreError: 241199:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7447.617579] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7448.550147] LustreError: 241197:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7448.589334] LustreError: 241197:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7455.086229] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7455.317114] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7460.343170] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7460.350209] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7461.723958] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7468.730633] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7472.583291] Lustre: Permanently reactivating lustre-MDT0001 [ 7472.622394] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7472.629730] Lustre: Skipped 2 previous similar messages [ 7475.273492] Lustre: lustre-MDT0001: Client 0d86ae9a-573f-4765-97e1-6520b61df409 (at 192.168.203.39@tcp) reconnecting [ 7480.957757] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7482.330604] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7487.540448] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7487.755724] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7491.157880] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7491.358974] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7493.400383] Lustre: Permanently deactivating lustre-MDT0001 [ 7493.410275] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7493.414423] Lustre: Skipped 2 previous similar messages [ 7500.385509] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7510.497881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7510.501490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7510.518937] Lustre: Skipped 3 previous similar messages [ 7510.553774] Lustre: Skipped 3 previous similar messages [ 7515.619138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7516.154524] Lustre: server umount lustre-MDT0000 complete [ 7520.738382] LustreError: 239349:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7520.752499] LustreError: 239349:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7520.853469] LustreError: 242904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446976 with bad export cookie 1598421693842736821 [ 7520.858198] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7520.861338] LustreError: 242904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7521.277353] Lustre: server umount lustre-MDT0001 complete [ 7530.675584] Lustre: server umount lustre-OST0000 complete [ 7542.528441] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:49:56 (1788446996) [ 7553.143304] Lustre: server umount lustre-OST0001 complete [ 7559.971510] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 7569.633773] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7603.218101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7603.506196] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7603.518903] Lustre: Skipped 2 previous similar messages [ 7603.544713] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7603.659951] Lustre: lustre-MDT0000: new disk, initializing [ 7603.809489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7603.827802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7608.218951] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7618.966408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7619.122217] Lustre: 247629:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7619.182389] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7619.186293] Lustre: Skipped 1 previous similar message [ 7619.263550] Lustre: lustre-MDT0001: new disk, initializing [ 7619.359397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7619.416946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7619.426442] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7623.761982] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7632.376543] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7637.867895] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7644.638599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7644.875455] Lustre: lustre-OST0000: new disk, initializing [ 7644.880291] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7644.886853] Lustre: 249599:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7644.967253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7651.204328] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7651.387101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7651.407179] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7651.616428] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7660.908346] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7663.017913] LustreError: 247635:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7671.605197] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7671.771119] Lustre: lustre-OST0001: new disk, initializing [ 7671.777843] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7671.785514] Lustre: 250713:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7673.023098] LustreError: 247635:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7673.049092] LustreError: 247637:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7677.864590] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7680.641213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7680.655909] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7680.747727] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7683.144371] LustreError: 247637:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7687.203956] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7688.787746] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7691.237839] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7691.238682] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7691.243752] LustreError: Skipped 1 previous similar message [ 7691.248335] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7691.248345] Lustre: Skipped 3 previous similar messages [ 7691.250261] Lustre: Skipped 1 previous similar message [ 7696.363239] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7696.372926] Lustre: Skipped 3 previous similar messages [ 7696.604974] Lustre: server umount lustre-OST0001 complete [ 7696.993443] LustreError: 249582:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7716.840262] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7716.858509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7716.874274] Lustre: Skipped 1 previous similar message [ 7720.603099] Lustre: server umount lustre-OST0000 complete [ 7727.091753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7727.092194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7727.107779] Lustre: Skipped 1 previous similar message [ 7727.108472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7727.123034] Lustre: Skipped 1 previous similar message [ 7730.707051] LustreError: 251993:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x2:0x0].0x0 (0000000010ba45d7) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 7732.192908] LustreError: 247636:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7732.224431] LustreError: 247636:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 7732.847642] Lustre: server umount lustre-MDT0000 complete [ 7736.581565] LustreError: 249598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447192 with bad export cookie 1598421693842739418 [ 7736.585771] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7736.594758] LustreError: 249598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7743.167234] Lustre: server umount lustre-MDT0001 complete [ 7756.006905] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7758.472738] Key type lgssc unregistered [ 7758.762827] LNet: 252745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7758.770315] LNetError: 252745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7758.786637] LNet: Removed LNI 192.168.203.139@tcp [ 7759.514592] Key type .llcrypt unregistered [ 7759.519299] Key type ._llcrypt unregistered [ 7799.720589] Key type ._llcrypt registered [ 7799.726437] Key type .llcrypt registered [ 7799.862642] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7800.980035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7800.991180] alg: No test for adler32 (adler32-zlib) [ 7802.017474] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7802.279165] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 7803.936343] Key type lgssc registered [ 7805.111430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7813.119907] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7813.144823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7814.420765] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7814.452622] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7814.588051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7819.391370] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7828.363114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7828.508067] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7828.580502] Lustre: 254952:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7828.872710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7833.335214] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7840.890819] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7846.713783] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7853.107495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7853.295693] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7853.518686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7859.652344] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7860.880509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7866.893939] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7917.536296] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7917.546168] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7917.559880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7922.145666] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7922.155713] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7922.173250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7923.663790] Lustre: server umount lustre-OST0000 complete [ 7928.426155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7940.820352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7941.291724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7942.340178] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7942.382529] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7947.508807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7947.543172] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7947.856881] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7955.521862] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7998.947646] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7998.957561] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7998.975882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7998.981446] Lustre: Skipped 1 previous similar message [ 8002.802912] Lustre: server umount lustre-OST0000 complete [ 8008.164988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8008.173132] LustreError: Skipped 1 previous similar message [ 8008.180381] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8008.190653] Lustre: Skipped 1 previous similar message [ 8008.203252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8008.209432] Lustre: Skipped 1 previous similar message [ 8014.276217] Lustre: server umount lustre-MDT0000 complete [ 8018.661392] LustreError: 254946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447474 with bad export cookie 17307266463881708489 [ 8018.665736] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8018.699395] LustreError: 254946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8019.031916] Lustre: server umount lustre-MDT0001 complete [ 8029.444175] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 8031.522801] Key type lgssc unregistered [ 8031.789821] LNet: 260373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8031.799189] LNetError: 260373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8032.872114] LNet: Removed LNI 192.168.203.139@tcp [ 8033.970779] Key type .llcrypt unregistered [ 8033.972703] Key type ._llcrypt unregistered [ 8057.217272] Key type ._llcrypt registered [ 8057.219704] Key type .llcrypt registered [ 8057.394161] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 8058.296137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8058.333205] alg: No test for adler32 (adler32-zlib) [ 8059.414336] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8059.656962] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 8061.343153] Key type lgssc registered [ 8062.661343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8069.212922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8069.241976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8070.821705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8075.006900] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8080.353552] LustreError: 261785:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8080.383410] LustreError: 261785:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8082.378752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8082.825425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8087.224308] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8094.566937] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8099.968917] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8107.556917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8107.871552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8115.357709] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8116.093226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8124.217317] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8152.033038] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8152.046125] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8152.061142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8153.056615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8153.062339] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8155.977714] Lustre: server umount lustre-OST0000 complete [ 8159.712421] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8159.728328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8159.744903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8159.751670] Lustre: Skipped 1 previous similar message [ 8163.297200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8163.300107] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8165.559523] Lustre: server umount lustre-MDT0000 complete [ 8168.416665] LustreError: 263026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8168.439983] LustreError: 263026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8169.126428] LustreError: 261767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447625 with bad export cookie 4281683757485615360 [ 8169.128616] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8169.136319] LustreError: 261767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8169.484266] Lustre: server umount lustre-MDT0001 complete [ 8179.709138] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 8182.599833] Key type lgssc unregistered [ 8182.931489] LNet: 266094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8182.939093] LNetError: 266094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8183.980464] LNet: Removed LNI 192.168.203.139@tcp [ 8184.850466] Key type .llcrypt unregistered [ 8184.852386] Key type ._llcrypt unregistered [ 8203.199954] Key type ._llcrypt registered [ 8203.202539] Key type .llcrypt registered [ 8203.299810] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 8203.974776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8203.982544] alg: No test for adler32 (adler32-zlib) [ 8204.928841] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8205.207434] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 8206.967653] Key type lgssc registered [ 8207.845589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8214.914733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8214.947843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8216.331524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8220.380654] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8226.787776] LustreError: 267327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8226.811055] LustreError: 267327:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8229.459880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8229.780046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8235.086426] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8243.738182] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8249.568821] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8256.677940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8257.058731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8261.186719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8263.951925] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8271.906946] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8286.688375] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8286.698232] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8286.724241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8289.760101] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8289.768834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8289.775480] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8289.805720] Lustre: Skipped 1 previous similar message [ 8292.453812] Lustre: server umount lustre-OST0000 complete [ 8296.418293] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8296.425645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8296.436285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8300.006114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8300.006824] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8302.608871] Lustre: server umount lustre-MDT0000 complete [ 8305.121100] LustreError: 267322:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8305.136918] LustreError: 267322:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8305.618089] LustreError: 269170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447761 with bad export cookie 16289023407733407811 [ 8305.636195] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8305.648462] LustreError: 269170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8305.981458] Lustre: server umount lustre-MDT0001 complete [ 8316.092842] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 8319.231587] Key type lgssc unregistered [ 8319.719310] LNet: 271003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8319.727751] LNetError: 271003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8320.808793] LNet: Removed LNI 192.168.203.139@tcp [ 8321.733229] Key type .llcrypt unregistered [ 8321.735381] Key type ._llcrypt unregistered [ 8348.290390] Key type ._llcrypt registered [ 8348.292829] Key type .llcrypt registered [ 8348.390383] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 8349.540708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8349.567088] alg: No test for adler32 (adler32-zlib) [ 8350.547569] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8350.782735] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 8352.447151] Key type lgssc registered [ 8353.244164] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8360.042545] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8360.063663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8361.412723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8364.924770] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8371.370046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8371.835635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8375.812352] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8385.368306] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8392.805542] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8400.263395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8400.518602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8406.891530] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8407.739161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8415.733742] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8443.881239] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8443.896050] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8443.917725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8447.974027] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8447.978651] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8447.995133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8449.349681] Lustre: server umount lustre-OST0000 complete [ 8453.605210] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8453.616257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8453.626135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8453.631462] Lustre: Skipped 1 previous similar message [ 8458.210178] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8458.210424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8459.673331] Lustre: server umount lustre-MDT0000 complete [ 8463.329447] LustreError: 272416:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8463.353103] LustreError: 272416:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8463.712823] LustreError: 272396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447919 with bad export cookie 4362373621526313850 [ 8463.713771] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8463.724223] LustreError: 272396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8464.144919] Lustre: server umount lustre-MDT0001 complete [ 8475.290396] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 8478.041946] Key type lgssc unregistered [ 8478.311075] LNet: 276721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8478.326156] LNetError: 276721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8478.344754] LNet: Removed LNI 192.168.203.139@tcp [ 8479.269765] Key type .llcrypt unregistered [ 8479.272711] Key type ._llcrypt unregistered [ 8496.053647] Key type ._llcrypt registered [ 8496.058352] Key type .llcrypt registered [ 8496.269373] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 8497.633313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8497.701157] alg: No test for adler32 (adler32-zlib) [ 8498.874163] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8499.238491] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 8501.007197] Key type lgssc registered [ 8502.229105] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8511.342586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8511.365027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8512.958455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8516.836394] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8522.210047] LustreError: 277955:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8522.237170] LustreError: 277955:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8525.754180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8526.174455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8531.534345] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8540.475191] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8548.667717] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8557.798528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8558.074510] LustreError: 279801:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8558.343462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8563.710380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8564.239146] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8572.193299] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8584.163398] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8584.175904] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8584.188439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8585.186555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8585.213574] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8588.613716] Lustre: server umount lustre-OST0000 complete [ 8592.864401] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8592.876092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8592.887887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8592.892966] Lustre: Skipped 1 previous similar message [ 8595.424860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8595.426181] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8598.216178] Lustre: server umount lustre-MDT0000 complete [ 8600.554519] LustreError: 277962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8600.577902] LustreError: 277962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8601.319733] LustreError: 277919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448057 with bad export cookie 68070291943350130 [ 8601.323831] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8601.337013] LustreError: 277919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8601.599398] Lustre: server umount lustre-MDT0001 complete [ 8609.827915] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 8611.897586] Key type lgssc unregistered [ 8612.142686] LNet: 281654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8612.148719] LNetError: 281654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8612.166592] LNet: Removed LNI 192.168.203.139@tcp [ 8612.959539] Key type .llcrypt unregistered [ 8612.963326] Key type ._llcrypt unregistered [ 8622.022271] Key type ._llcrypt registered [ 8622.025028] Key type .llcrypt registered [ 8622.119979] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8636.796340] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 8650.856393] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 8652.366384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8652.570670] alg: No test for adler32 (adler32-zlib) [ 8653.771848] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8654.015919] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 8655.736069] Key type lgssc registered [ 8657.466872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8702.437688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8702.468508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8703.833408] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8703.888418] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8704.025047] Lustre: lustre-MDT0000: new disk, initializing [ 8704.185672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8704.212867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8709.911561] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8721.617702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8721.757799] Lustre: 285186:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8721.812715] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8721.821132] Lustre: Skipped 1 previous similar message [ 8721.911949] Lustre: lustre-MDT0001: new disk, initializing [ 8722.032656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8722.074102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8722.084848] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8727.882554] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8738.339961] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8745.023034] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8754.133730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8754.746743] Lustre: lustre-OST0000: new disk, initializing [ 8754.751888] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8754.761947] Lustre: 287153:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8754.908576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8761.787364] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8762.445356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8762.460413] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8762.576141] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8771.003039] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8776.092417] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8776.349289] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8781.347696] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8781.570930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8788.457846] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8788.474571] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8788.489839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8788.495771] Lustre: Skipped 1 previous similar message [ 8789.850700] Lustre: server umount lustre-OST0000 complete [ 8793.568235] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8793.575939] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8793.577478] LustreError: Skipped 1 previous similar message [ 8793.580969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8793.610990] Lustre: Skipped 2 previous similar messages [ 8798.698216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8798.701943] Lustre: Skipped 1 previous similar message [ 8800.010434] Lustre: server umount lustre-MDT0000 complete [ 8804.364861] LustreError: 285177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448260 with bad export cookie 10497687160263663321 [ 8804.370805] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8804.383317] LustreError: 285177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8804.741413] Lustre: server umount lustre-MDT0001 complete [ 8813.405466] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:11:07 (1788448267) [ 8820.548741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8821.250415] LustreError: 289569:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8821.269225] LustreError: 289569:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8821.384507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8826.314398] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8826.857072] LustreError: 289570:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8831.977849] LustreError: 289569:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8835.655449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8835.928975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8840.163317] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8848.196434] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8855.608604] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8863.390985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8863.711404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8870.042347] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8878.094919] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8890.349980] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8890.357677] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8890.374902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8890.406972] Lustre: Skipped 1 previous similar message [ 8894.305870] Lustre: server umount lustre-OST0000 complete [ 8900.577988] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8900.580052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8900.592889] Lustre: Skipped 1 previous similar message [ 8900.608666] Lustre: Skipped 1 previous similar message [ 8910.819934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8910.823944] Lustre: Skipped 3 previous similar messages [ 8913.375494] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8913.585355] Lustre: server umount lustre-MDT0000 complete [ 8915.937615] LustreError: 289566:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8915.952758] LustreError: 289566:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8917.905646] LustreError: 289551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448373 with bad export cookie 10497687160263664203 [ 8917.912708] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8917.932235] LustreError: 289551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8918.396621] Lustre: server umount lustre-MDT0001 complete [ 8930.488797] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 8933.413563] Key type lgssc unregistered [ 8933.731955] LNet: 293106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8933.740888] LNetError: 293106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8933.757647] LNet: Removed LNI 192.168.203.139@tcp [ 8934.495920] Key type .llcrypt unregistered [ 8934.499682] Key type ._llcrypt unregistered [ 8963.793736] Key type ._llcrypt registered [ 8963.795714] Key type .llcrypt registered [ 8963.919320] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 8979.664755] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 8980.939282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8981.144500] alg: No test for adler32 (adler32-zlib) [ 8982.281419] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8982.638458] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 8984.315668] Key type lgssc registered [ 8985.628850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9041.705470] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9041.781174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9043.061786] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9043.088333] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9043.154175] Lustre: lustre-MDT0000: new disk, initializing [ 9043.236876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9043.258653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9048.060865] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9059.535081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9059.745630] Lustre: 296932:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9059.925686] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9059.936470] Lustre: Skipped 1 previous similar message [ 9060.025034] Lustre: lustre-MDT0001: new disk, initializing [ 9060.151122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9060.182258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9060.195716] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9065.463220] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9076.259509] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9082.844565] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9090.396696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9090.640656] Lustre: lustre-OST2710: new disk, initializing [ 9090.644773] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9090.653076] Lustre: 298901:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9090.778530] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9096.943488] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9097.791182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9097.805087] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9097.898313] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9107.220647] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9114.703198] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9114.981663] Lustre: lustre-OST03e8: new disk, initializing [ 9114.986888] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9114.996985] Lustre: 299965:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9115.116217] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9121.680958] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9122.902955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9122.912570] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9123.046811] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9129.300827] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9136.006063] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9136.192573] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9140.389849] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9140.562044] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9145.458419] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9145.642101] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9150.099588] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9150.310750] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9157.138225] Lustre: server umount lustre-MDT0000 complete [ 9157.600550] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9157.616995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9159.146076] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9159.161375] Lustre: Skipped 1 previous similar message [ 9160.903876] LustreError: 298911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448616 with bad export cookie 311439313642037723 [ 9160.908153] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9160.915447] LustreError: 298911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9161.166320] Lustre: server umount lustre-MDT0001 complete [ 9175.100058] Lustre: server umount lustre-OST2710 complete [ 9188.266682] Lustre: server umount lustre-OST03e8 complete [ 9206.261287] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 9216.288191] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 9252.121436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9252.476797] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9252.546899] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9252.662976] Lustre: lustre-MDT0000: new disk, initializing [ 9252.827903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9252.854766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9257.461272] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9267.715342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9267.771806] Lustre: 305215:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9267.807564] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9267.810593] Lustre: Skipped 1 previous similar message [ 9267.856270] Lustre: lustre-MDT0001: new disk, initializing [ 9267.917292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9267.945685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9267.955131] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9273.118089] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9285.116452] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9291.543864] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9298.371641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9298.552659] Lustre: lustre-OST0000: new disk, initializing [ 9298.556287] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9298.561581] Lustre: 307185:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9298.638317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9304.015606] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9307.724432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9307.736443] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9307.850434] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9313.717645] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9319.227314] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9319.503655] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9324.032253] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9324.227588] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9328.616875] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9328.617194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9328.622577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9328.630042] LustreError: Skipped 1 previous similar message [ 9328.642472] Lustre: Skipped 2 previous similar messages [ 9333.740749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9333.749269] Lustre: Skipped 2 previous similar messages [ 9338.854938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9338.869935] Lustre: Skipped 1 previous similar message [ 9340.383252] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9340.544810] Lustre: server umount lustre-OST0000 complete [ 9343.971430] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9343.981730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9343.986468] Lustre: Skipped 1 previous similar message [ 9349.089239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9349.098863] Lustre: Skipped 2 previous similar messages [ 9358.305746] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9358.529839] Lustre: server umount lustre-MDT0000 complete [ 9359.329512] LustreError: 305221:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9359.370083] LustreError: 305221:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 9361.932884] LustreError: 305208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448817 with bad export cookie 311439313642039403 [ 9361.937662] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9361.946867] LustreError: 305208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9362.348913] Lustre: server umount lustre-MDT0001 complete [ 9370.041044] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:20:24 (1788448824) [ 9371.324995] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9372.851487] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:20:27 (1788448827) [ 9376.100490] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 9400.396182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9400.698447] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9400.725272] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9400.916408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9404.636575] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9412.494393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9412.550139] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9412.568953] Lustre: 311042:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9416.430605] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9436.840901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9437.043632] Lustre: Denying initial registration attempt from nid 192.168.203.139@tcp, specified as failover [ 9437.065221] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9437.077607] LustreError: 312499:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -99 [ 9437.087976] LustreError: 312499:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9437.100960] LustreError: 312499:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9437.195427] Lustre: server umount lustre-OST0000 complete [ 9437.198585] LustreError: 312499:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9444.833517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9444.868530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9444.876293] Lustre: Skipped 3 previous similar messages [ 9450.295438] Lustre: server umount lustre-MDT0000 complete [ 9453.875356] LustreError: 311035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448909 with bad export cookie 311439313642040285 [ 9453.881025] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9453.885987] LustreError: 311035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9454.278448] Lustre: server umount lustre-MDT0001 complete [ 9464.171216] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 9466.728369] Key type lgssc unregistered [ 9467.028272] LNet: 313617:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9467.037603] LNetError: 313617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9467.058717] LNet: Removed LNI 192.168.203.139@tcp [ 9468.016408] Key type .llcrypt unregistered [ 9468.020991] Key type ._llcrypt unregistered [ 9479.549879] Key type ._llcrypt registered [ 9479.553739] Key type .llcrypt registered [ 9479.658212] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 9480.554076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9480.733302] alg: No test for adler32 (adler32-zlib) [ 9481.799684] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9482.001912] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 9483.687157] Key type lgssc registered [ 9484.953517] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9534.459338] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 9544.328970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9544.363458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9545.769553] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9545.817759] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9546.005497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9550.480363] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9559.520334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9559.586139] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9559.608907] Lustre: 316382:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9559.844247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9564.119721] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9572.158781] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9577.893574] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9598.276864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9598.649856] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9598.964772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9607.008265] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9616.597420] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9622.501529] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9622.514113] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9622.537799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9623.034200] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9623.053266] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9623.079983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9623.099162] Lustre: Skipped 1 previous similar message [ 9628.150537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9633.249045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9633.251403] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 9633.272386] Lustre: Skipped 1 previous similar message [ 9633.716322] Lustre: server umount lustre-OST0000 complete [ 9638.371550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9638.385403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9638.392786] Lustre: Skipped 1 previous similar message [ 9638.423032] Lustre: Skipped 2 previous similar messages [ 9648.609084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9648.627578] Lustre: Skipped 2 previous similar messages [ 9652.191186] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9652.331940] Lustre: server umount lustre-MDT0000 complete [ 9657.035791] LustreError: 316372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449112 with bad export cookie 10674917721653379815 [ 9657.037156] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9657.045279] LustreError: 316372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9657.409651] Lustre: server umount lustre-MDT0001 complete [ 9671.508559] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 9674.147449] Key type lgssc unregistered [ 9674.489225] LNet: 319956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9674.500836] LNetError: 319956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9674.542847] LNet: Removed LNI 192.168.203.139@tcp [ 9675.513451] Key type .llcrypt unregistered [ 9675.517574] Key type ._llcrypt unregistered [ 9699.492589] Key type ._llcrypt registered [ 9699.494421] Key type .llcrypt registered [ 9699.601222] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 9700.489621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9700.601847] alg: No test for adler32 (adler32-zlib) [ 9701.632490] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9701.877626] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 9703.567230] Key type lgssc registered [ 9704.701771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9714.483792] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9714.508705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9716.074700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9721.585595] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9725.413860] LustreError: 321369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9725.451232] LustreError: 321369:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9730.533939] LustreError: 321368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9730.781081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9731.122170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9735.246578] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9743.295935] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9749.558518] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9759.024033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9759.396575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9766.358023] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9774.566163] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9791.613550] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.39@tcp (stopping) [ 9792.485770] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9792.491344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9792.511453] Lustre: Skipped 1 previous similar message [ 9792.528129] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9794.913643] Lustre: server umount lustre-MDT0000 complete [ 9796.696707] LustreError: 321363:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9798.704409] LustreError: 323978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449254 with bad export cookie 7547655730865744525 [ 9798.706183] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9798.714570] LustreError: 323978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9801.820627] LustreError: 321363:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9801.847167] LustreError: 321363:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9801.886022] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.39@tcp (stopping) [ 9801.895190] Lustre: Skipped 2 previous similar messages [ 9802.726160] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9805.350596] Lustre: server umount lustre-MDT0001 complete [ 9813.051501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9823.091863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9843.679857] LustreError: 320676:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ae8ec883 x1875324863297536/t0(0) o250->MGC192.168.203.139@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9843.940934] LustreError: 323200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9843.953688] LustreError: 323200:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9844.025175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9844.083565] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9848.109684] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9853.413731] LustreError: 324989:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9853.441645] LustreError: 324989:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9856.095691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9856.369959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9858.430302] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9861.205235] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9869.151418] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9870.737805] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9876.357965] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9877.922588] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9884.134272] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9884.145931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9884.146259] Lustre: Skipped 1 previous similar message [ 9884.154837] Lustre: Skipped 3 previous similar messages [ 9888.989467] Lustre: server umount lustre-OST0000 complete [ 9892.334813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9892.346537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9892.360097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9898.641693] Lustre: server umount lustre-MDT0000 complete [ 9899.496131] LustreError: 324994:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9901.837767] LustreError: 321349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449357 with bad export cookie 7547655730865752365 [ 9901.843304] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9901.853417] LustreError: 321349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9902.156804] Lustre: server umount lustre-MDT0001 complete [ 9912.265150] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 9914.842457] Key type lgssc unregistered [ 9915.208811] LNet: 327664:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9915.217429] LNetError: 327664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9915.236507] LNet: Removed LNI 192.168.203.139@tcp [ 9916.093186] Key type .llcrypt unregistered [ 9916.102047] Key type ._llcrypt unregistered [ 9941.756659] Key type ._llcrypt registered [ 9941.760301] Key type .llcrypt registered [ 9941.896763] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 9943.144538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9943.190450] alg: No test for adler32 (adler32-zlib) [ 9944.490717] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9944.939979] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 9946.720830] Key type lgssc registered [ 9948.335521] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9958.251330] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9958.270317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9959.754575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9965.196476] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9969.127080] LustreError: 329082:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9969.143447] LustreError: 329082:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9974.243938] LustreError: 329083:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9974.481588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9974.867463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9979.075589] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9986.628624] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9993.191651] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9997.623746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9997.630303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9997.645356] Lustre: Skipped 1 previous similar message [ 9997.662355] Lustre: Skipped 1 previous similar message [ 9999.716672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9999.728717] Lustre: Skipped 1 previous similar message [10004.091918] Lustre: server umount lustre-MDT0000 complete [10004.971171] LustreError: 329082:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10004.999667] LustreError: 329082:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10008.760856] LustreError: 329064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449464 with bad export cookie 3866698054706484149 [10008.773040] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10008.778772] LustreError: 329064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10009.349515] Lustre: server umount lustre-MDT0001 complete [10025.157869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10025.458796] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10025.512145] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10025.735445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10029.896544] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10039.451485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10039.624147] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10039.683162] Lustre: 331670:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10039.802951] LustreError: 331681:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10040.027607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10045.093357] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10053.155858] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10059.154533] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10069.166185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10069.380210] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10069.391715] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10069.405554] LustreError: 333553:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [10069.410742] LustreError: 333553:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10069.415932] LustreError: 333553:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10069.487692] Lustre: server umount lustre-OST0000 complete [10069.491414] LustreError: 333553:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10079.216086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10079.478459] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10079.701032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10085.871316] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10088.075525] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10093.004392] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10101.343624] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10101.444402] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10101.533730] Lustre: lustre-OST0001: new disk, initializing [10101.538090] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10101.548046] Lustre: 335095:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10101.679584] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10108.371574] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10109.043582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10109.065717] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10109.166589] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10116.366932] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10119.653349] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10119.653750] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10119.654809] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10119.661041] Lustre: Skipped 1 previous similar message [10124.775694] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10124.782844] Lustre: Skipped 2 previous similar messages [10125.580735] Lustre: server umount lustre-OST0001 complete [10129.379866] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10129.388603] LustreError: Skipped 1 previous similar message [10129.395218] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10129.416969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10129.897661] LustreError: 334507:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10129.919360] LustreError: 334507:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10135.579985] Lustre: server umount lustre-OST0000 complete [10140.130253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10140.136322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10140.148118] Lustre: Skipped 2 previous similar messages [10140.165935] Lustre: Skipped 4 previous similar messages [10145.222679] Lustre: server umount lustre-MDT0000 complete [10145.254556] LustreError: 331682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10145.283127] LustreError: 331682:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10150.241417] LustreError: 331662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449606 with bad export cookie 3866698054706484702 [10150.242792] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10150.252192] LustreError: 331662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10150.885495] Lustre: server umount lustre-MDT0001 complete [10161.435651] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [10164.070780] Key type lgssc unregistered [10164.327746] LNet: 337051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10164.335121] LNetError: 337051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10164.353967] LNet: Removed LNI 192.168.203.139@tcp [10165.249767] Key type .llcrypt unregistered [10165.251839] Key type ._llcrypt unregistered [10240.859840] Key type ._llcrypt registered [10240.861871] Key type .llcrypt registered [10241.025695] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [10259.551962] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [10260.425961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10260.439532] alg: No test for adler32 (adler32-zlib) [10261.476805] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10261.728269] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [10263.431419] Key type lgssc registered [10264.635996] Lustre: Echo OBD driver; http://www.lustre.org/ [10302.775844] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10302.824390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10304.151239] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10304.222604] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10304.354429] Lustre: lustre-MDT0000: new disk, initializing [10304.504496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10304.537954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10309.362143] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10320.422181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10320.489717] Lustre: 341987:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10320.526580] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10320.531569] Lustre: Skipped 1 previous similar message [10320.586521] Lustre: lustre-MDT0001: new disk, initializing [10320.667466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10320.704690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10320.724159] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10325.469352] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10335.797727] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10341.719802] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10348.051950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10348.327365] Lustre: lustre-OST0000: new disk, initializing [10348.334867] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10348.343157] Lustre: 343954:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10348.426942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10354.661696] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10357.834062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10357.840368] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10357.954216] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10363.854176] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10367.860704] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10368.130865] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10373.237847] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10373.537286] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10378.721879] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10378.723694] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10378.732715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10378.741663] Lustre: Skipped 1 previous similar message [10378.759966] Lustre: Skipped 1 previous similar message [10381.956574] Lustre: server umount lustre-OST0000 complete [10387.940075] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10387.955795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10387.990076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10388.969410] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10388.992815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10389.003712] Lustre: Skipped 1 previous similar message [10393.888709] Lustre: server umount lustre-MDT0000 complete [10397.803457] LustreError: 341978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449853 with bad export cookie 865249347511656550 [10397.808769] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10397.816433] LustreError: 341978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10398.169360] Lustre: server umount lustre-MDT0001 complete [10407.046743] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:37:41 (1788449861) [10415.246818] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:37:49 (1788449869) [10423.857531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10424.512326] LustreError: 346708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10424.535481] LustreError: 346708:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10424.646684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10429.851211] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10429.930950] LustreError: 346709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10435.046371] LustreError: 346708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10439.314498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10439.596478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10443.641775] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10453.018949] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10459.578986] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10467.429305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10467.783133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10474.499724] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10482.782465] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10490.923585] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10493.340307] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10495.420896] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10497.853624] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10499.815284] Lustre: Failing over lustre-MDT0000 [10500.104542] Lustre: server umount lustre-MDT0000 complete [10500.214881] LustreError: 348660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10501.091609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10501.104866] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10504.675033] LustreError: 346704:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10504.703137] LustreError: 346704:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10514.914070] LustreError: 346705:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10514.941578] LustreError: 346705:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10520.227432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10520.379500] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10520.641664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10520.699282] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10525.667181] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10525.697110] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10525.882561] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10525.901847] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10525.966593] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10536.434437] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10538.283645] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10543.026936] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10545.365996] Lustre: Failing over lustre-MDT0000 [10545.697309] Lustre: server umount lustre-MDT0000 complete [10546.148896] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10546.156447] LustreError: 348660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10546.160567] Lustre: Skipped 4 previous similar messages [10546.200242] LustreError: 348660:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10562.399275] Lustre: 340211:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788450002/real 1788450002] req@00000000652e4c22 x1875325450542336/t0(0) o400->MGC192.168.203.139@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788450018 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10562.437980] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10565.650468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10572.774732] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc01fbb280922afb [10572.786757] Lustre: MGC192.168.203.139@tcp: Connection restored to 0@lo (at 0@lo) [10572.795576] Lustre: Skipped 2 previous similar messages [10572.887115] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.39@tcp (not set up) [10573.073285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10573.132108] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10577.978428] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10578.024514] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10578.413102] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10578.437603] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10578.497390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10588.503666] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10590.195468] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10598.880284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10598.885620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10598.897910] Lustre: Skipped 2 previous similar messages [10598.898141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10598.915643] Lustre: Skipped 2 previous similar messages [10602.825526] Lustre: server umount lustre-MDT0000 complete [10604.006541] LustreError: 346709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10604.038180] LustreError: 346709:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [10607.157363] LustreError: 346688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450063 with bad export cookie 865249347511659259 [10607.162130] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10607.170296] LustreError: 346688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10607.572565] Lustre: server umount lustre-MDT0001 complete [10621.729789] Lustre: server umount lustre-OST0000 complete [10640.496470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10641.220373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10646.981435] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10658.371328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10663.308753] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10670.858455] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10677.666600] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10685.410783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10686.780351] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10692.765889] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10701.384229] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10712.548922] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10712.559867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10712.585942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10717.961424] Lustre: server umount lustre-OST0000 complete [10728.476669] Lustre: server umount lustre-MDT0000 complete [10728.931688] LustreError: 353060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10728.946219] LustreError: 353060:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [10732.697606] LustreError: 353040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450188 with bad export cookie 865249347511660008 [10732.699533] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10732.707786] LustreError: 353040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10733.163569] Lustre: server umount lustre-MDT0001 complete [10746.008585] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [10748.914466] Key type lgssc unregistered [10749.280457] LNet: 356593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10749.297567] LNetError: 356593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10749.312505] LNet: Removed LNI 192.168.203.139@tcp [10750.204291] Key type .llcrypt unregistered [10750.209075] Key type ._llcrypt unregistered [10778.035346] Key type ._llcrypt registered [10778.038384] Key type .llcrypt registered [10778.205140] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [10779.127562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10779.140131] alg: No test for adler32 (adler32-zlib) [10780.282125] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10780.532983] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [10782.207267] Key type lgssc registered [10783.357292] Lustre: Echo OBD driver; http://www.lustre.org/ [10792.184786] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10792.200699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10793.614310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10798.354554] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10803.174605] LustreError: 358106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10803.214741] LustreError: 358106:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10807.651362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10808.110835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10812.784234] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10821.551298] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10827.286451] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10834.471234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10834.831509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10838.970718] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10840.700909] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10848.906336] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10864.608537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10864.616217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10864.640015] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10865.686767] Lustre: server umount lustre-MDT0000 complete [10867.680708] LustreError: 359330:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10867.680868] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10867.696724] LustreError: 359330:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10867.712608] Lustre: Skipped 1 previous similar message [10868.998087] LustreError: 358087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450324 with bad export cookie 4289607860927071819 [10869.006308] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10869.342637] Lustre: server umount lustre-MDT0001 complete [10883.188559] Lustre: server umount lustre-OST0000 complete [10896.076429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10896.347549] LustreError: 361838:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10896.367125] LustreError: 361838:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10896.407251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10899.967372] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10901.480571] LustreError: 361839:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10906.593402] LustreError: 361838:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10907.644962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10908.012517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10912.481758] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10920.096809] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10925.783456] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10932.812778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10933.174878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10938.833360] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10941.367196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10946.428589] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10957.812650] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10957.824728] Lustre: Skipped 2 previous similar messages [10957.825166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10957.841636] Lustre: Skipped 2 previous similar messages [10962.915706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10962.929513] Lustre: Skipped 2 previous similar messages [10963.727896] Lustre: server umount lustre-MDT0000 complete [10967.514673] LustreError: 364518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450423 with bad export cookie 4289607860927072932 [10967.516739] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10967.536339] LustreError: 364518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10967.905534] Lustre: server umount lustre-MDT0001 complete [10982.842609] Lustre: server umount lustre-OST0000 complete [10997.408791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10997.842870] LustreError: 365747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10997.926286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11002.459119] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11010.958476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11015.710362] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11022.464758] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11028.017475] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11034.070561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11034.384721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11034.400178] Lustre: Skipped 1 previous similar message [11040.214894] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11042.604349] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11047.816389] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11053.049986] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11053.054976] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11053.078880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11053.086109] Lustre: Skipped 1 previous similar message [11058.151851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11058.571310] Lustre: server umount lustre-OST0000 complete [11062.774705] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11062.797916] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11062.829921] Lustre: Skipped 1 previous similar message [11068.394510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11068.406945] Lustre: Skipped 5 previous similar messages [11068.558893] Lustre: server umount lustre-MDT0000 complete [11072.168274] LustreError: 367589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450528 with bad export cookie 4289607860927073947 [11072.172054] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11072.188260] LustreError: 367589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11072.470597] Lustre: server umount lustre-MDT0001 complete [11082.332139] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11085.035619] Key type lgssc unregistered [11085.255380] LNet: 369276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11085.269040] LNetError: 369276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11085.291109] LNet: Removed LNI 192.168.203.139@tcp [11086.086536] Key type .llcrypt unregistered [11086.089096] Key type ._llcrypt unregistered [11111.009879] Key type ._llcrypt registered [11111.012903] Key type .llcrypt registered [11111.119792] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [11112.150326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11112.186713] alg: No test for adler32 (adler32-zlib) [11113.237385] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11113.411930] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [11115.063168] Key type lgssc registered [11116.213658] Lustre: Echo OBD driver; http://www.lustre.org/ [11124.416977] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11124.447838] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11124.456407] LustreError: 370716:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11124.478718] LustreError: 370716:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11124.489253] LustreError: 370716:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11124.499305] LustreError: 370716:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11124.512058] LustreError: 370716:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11131.983242] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11131.990663] LustreError: 371113:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11132.013706] LustreError: 371113:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11132.023600] LustreError: 371113:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11132.033031] LustreError: 371113:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11132.041264] LustreError: 371113:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11151.745596] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11154.854568] Key type lgssc unregistered [11155.132628] LNet: 372116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11155.149071] LNetError: 372116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11155.165140] LNet: Removed LNI 192.168.203.139@tcp [11156.171306] Key type .llcrypt unregistered [11156.178298] Key type ._llcrypt unregistered [11176.286670] Key type ._llcrypt registered [11176.288242] Key type .llcrypt registered [11176.416592] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [11192.145953] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [11193.507043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11193.556752] alg: No test for adler32 (adler32-zlib) [11194.567763] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11194.826146] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [11196.519177] Key type lgssc registered [11197.614659] Lustre: Echo OBD driver; http://www.lustre.org/ [11236.334050] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11236.385244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11237.696595] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11237.750566] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11237.898540] Lustre: lustre-MDT0000: new disk, initializing [11238.041190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11238.071965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11243.405445] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11253.931770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11254.057485] Lustre: 375270:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11254.095072] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11254.100326] Lustre: Skipped 1 previous similar message [11254.158932] Lustre: lustre-MDT0001: new disk, initializing [11254.243661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11254.284381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11254.301765] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11258.841988] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11270.099948] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11277.484078] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11285.719501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11285.990593] Lustre: lustre-OST0000: new disk, initializing [11285.994295] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11286.000171] Lustre: 377237:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11286.077293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11292.563838] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11295.838964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11295.848911] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11296.003232] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11302.223684] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11307.363771] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11307.567482] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11311.757488] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11311.975539] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11316.711416] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11316.726333] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11316.733707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11316.754731] Lustre: Skipped 1 previous similar message [11320.479196] Lustre: server umount lustre-OST0000 complete [11325.920834] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11325.931145] LustreError: Skipped 1 previous similar message [11325.933291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11325.951492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11325.955525] Lustre: Skipped 1 previous similar message [11326.946949] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11326.957955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11326.959924] Lustre: Skipped 1 previous similar message [11330.740376] Lustre: server umount lustre-MDT0000 complete [11334.289384] LustreError: 375260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450790 with bad export cookie 3990344954713110993 [11334.296720] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11334.301174] LustreError: 375260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11334.639484] Lustre: server umount lustre-MDT0001 complete [11342.735715] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11344.379603] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 11:53:18 (1788450798) [11350.123221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11350.566242] LustreError: 379697:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11350.583211] LustreError: 379697:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11350.663870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11354.889801] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11356.131601] LustreError: 379698:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11360.228058] LustreError: 379697:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11362.159026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11362.416513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11366.553565] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11373.529352] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11379.325848] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11385.279856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11385.631107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11390.670830] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11397.380736] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11403.434552] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11403.555838] Lustre: lustre-OST0001: new disk, initializing [11403.559294] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11403.564702] Lustre: 382532:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11403.642123] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11409.295241] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11411.007075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11411.021421] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11411.100860] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11417.877572] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11421.668422] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11421.672368] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11421.677701] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11421.683914] LustreError: Skipped 1 previous similar message [11421.714138] Lustre: Skipped 1 previous similar message [11426.796600] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11426.805021] Lustre: Skipped 3 previous similar messages [11427.742759] Lustre: server umount lustre-OST0001 complete [11429.980957] LustreError: 381518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11437.030606] LustreError: 383206:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11437.053131] LustreError: 383206:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11444.194656] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11444.200477] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11444.210459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11447.270378] LustreError: 381518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11447.285816] LustreError: 381518:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11450.414762] Lustre: server umount lustre-OST0000 complete [11454.432722] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11454.446375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11454.464880] Lustre: Skipped 1 previous similar message [11460.763796] Lustre: server umount lustre-MDT0000 complete [11465.692533] LustreError: 379678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450921 with bad export cookie 3990344954713111875 [11465.693431] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11465.727067] LustreError: 379678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11472.408920] Lustre: server umount lustre-MDT0001 complete [11484.203528] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11486.418414] Key type lgssc unregistered [11486.656237] LNet: 384518:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11486.667761] LNetError: 384518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11486.684995] LNet: Removed LNI 192.168.203.139@tcp [11487.444340] Key type .llcrypt unregistered [11487.447964] Key type ._llcrypt unregistered [11533.753730] Key type ._llcrypt registered [11533.757074] Key type .llcrypt registered [11533.840274] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [11534.506685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11534.534141] alg: No test for adler32 (adler32-zlib) [11535.643814] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11536.050540] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [11537.767206] Key type lgssc registered [11539.038684] Lustre: Echo OBD driver; http://www.lustre.org/ [11547.800404] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11547.825402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11549.104295] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11549.139632] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11549.348109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11553.711849] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11563.200212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11563.312987] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11563.359828] Lustre: 387335:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11563.588041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11569.978919] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11577.172395] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11582.457860] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11588.577478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11588.605425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11590.808641] Lustre: server umount lustre-MDT0000 complete [11594.913632] LustreError: 387327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451050 with bad export cookie 15162320689892733200 [11594.917313] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11594.929342] LustreError: 387327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11595.255980] Lustre: server umount lustre-MDT0001 complete [11605.241788] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 11:57:39 (1788451059) [11612.093867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11612.530686] LustreError: 389848:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11612.548697] LustreError: 389848:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11612.589962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11617.509379] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11617.763953] LustreError: 389849:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11622.884562] LustreError: 389848:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11626.539454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11626.910333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11632.041121] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11639.902467] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11645.119868] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11650.891430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11651.064644] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11651.261892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11656.538605] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11664.144930] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11672.092648] Lustre: Permanently deactivating lustre-OST0000 [11672.120161] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11679.788452] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11681.306994] LustreError: 392837:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11681.315843] LustreError: 392837:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11681.321850] LustreError: 392837:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11684.184956] LustreError: 392885:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11684.190899] LustreError: 392885:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11692.275168] Lustre: server umount lustre-OST0000 complete [11695.066656] LustreError: 393134:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11695.071524] LustreError: 393134:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11697.127750] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11697.143784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11697.146358] Lustre: Skipped 2 previous similar messages [11697.159751] Lustre: Skipped 2 previous similar messages [11702.240991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11702.261314] Lustre: Skipped 1 previous similar message [11707.361460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11707.369973] Lustre: Skipped 1 previous similar message [11707.453205] Lustre: server umount lustre-MDT0000 complete [11711.000416] LustreError: 391680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451166 with bad export cookie 15162320689892733781 [11711.015614] LustreError: 391680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11711.028259] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11720.366974] Lustre: server umount lustre-MDT0001 complete [11730.009904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11734.860537] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11737.560856] LustreError: 394293:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11743.249880] Lustre: 394632:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.139@tcp [11745.412070] Lustre: server umount MGS complete [11757.866523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11758.269719] LustreError: 395357:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11758.293425] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11758.297932] Lustre: Skipped 1 previous similar message [11758.335448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11763.100968] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11763.689842] LustreError: 395358:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11771.445756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11771.742496] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11776.406148] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11784.228426] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11789.998962] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11792.183661] Lustre: Permanently reactivating lustre-OST0000 [11792.195072] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11792.206023] Lustre: Skipped 2 previous similar messages [11823.075019] LustreError: 397235:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11830.139349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11830.565604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11830.571778] Lustre: Skipped 1 previous similar message [11836.205110] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11843.038671] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11848.160912] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11848.171217] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11848.189413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11854.310413] LustreError: 398168:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11854.337969] LustreError: 398168:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11854.474351] Lustre: server umount lustre-OST0000 complete [11858.928409] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11858.939552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11858.957890] Lustre: Skipped 1 previous similar message [11858.966827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11858.975697] Lustre: Skipped 2 previous similar messages [11864.491735] Lustre: server umount lustre-MDT0000 complete [11868.382156] LustreError: 395339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451324 with bad export cookie 15162320689892734831 [11868.388734] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11868.397873] LustreError: 395339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11868.783784] Lustre: server umount lustre-MDT0001 complete [11879.743828] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11882.671988] Key type lgssc unregistered [11883.042182] LNet: 399876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11883.054337] LNetError: 399876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11883.063916] LNet: Removed LNI 192.168.203.139@tcp [11884.003504] Key type .llcrypt unregistered [11884.006251] Key type ._llcrypt unregistered [11901.607739] Key type ._llcrypt registered [11901.609535] Key type .llcrypt registered [11901.709679] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [11916.557184] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [11917.878140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11918.123517] alg: No test for adler32 (adler32-zlib) [11919.213956] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11919.636228] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [11921.383143] Key type lgssc registered [11922.716265] Lustre: Echo OBD driver; http://www.lustre.org/ [11960.788632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11960.845241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11962.145536] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11962.198241] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11962.301521] Lustre: lustre-MDT0000: new disk, initializing [11962.367538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11962.384467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11967.238715] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11977.664537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11977.845314] Lustre: 403028:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11978.016943] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11978.033709] Lustre: Skipped 1 previous similar message [11978.170691] Lustre: lustre-MDT0001: new disk, initializing [11978.297019] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11978.345347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11978.370076] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11984.215775] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11994.551350] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12001.461754] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12009.122750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12009.377196] Lustre: lustre-OST0000: new disk, initializing [12009.383480] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12009.395743] Lustre: 404997:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12009.511517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12015.775715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12015.812414] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12016.087286] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12016.205148] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12027.230858] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12032.702532] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12032.950316] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12037.786346] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12037.989328] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12041.697414] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12041.706695] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12041.710415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12041.724573] Lustre: Skipped 1 previous similar message [12041.748924] Lustre: Skipped 1 previous similar message [12046.285476] Lustre: server umount lustre-OST0000 complete [12051.937603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12051.951514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12051.960533] Lustre: Skipped 1 previous similar message [12056.376902] Lustre: server umount lustre-MDT0000 complete [12060.172312] LustreError: 404996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451516 with bad export cookie 9835483862498456960 [12060.179246] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12060.192403] LustreError: 404996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12060.604567] Lustre: server umount lustre-MDT0001 complete [12069.022243] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12070.891706] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 12:05:25 (1788451525) [12077.947463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12078.396890] LustreError: 407458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12078.416736] LustreError: 407458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12078.471371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12083.468318] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12083.686542] LustreError: 407459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12088.802756] LustreError: 407458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12092.803287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12093.198609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12098.009784] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12105.343724] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12111.270408] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12118.132922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12118.632992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12124.911954] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12132.919937] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12134.830393] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12137.576466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12144.615509] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12144.622375] LustreError: Skipped 1 previous similar message [12144.625573] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12144.637045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12144.659882] Lustre: Skipped 2 previous similar messages [12149.031612] Lustre: server umount lustre-OST0000 complete [12154.848435] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12154.852268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12154.857237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12154.869373] Lustre: Skipped 1 previous similar message [12154.886385] Lustre: Skipped 2 previous similar messages [12159.969075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12159.978024] Lustre: Skipped 1 previous similar message [12167.135110] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12169.184876] LustreError: 408186:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12169.217162] LustreError: 408186:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12169.373410] Lustre: server umount lustre-MDT0000 complete [12173.170955] LustreError: 407440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451629 with bad export cookie 9835483862498457842 [12173.172759] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12173.188858] LustreError: 407440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12173.580172] Lustre: server umount lustre-MDT0001 complete [12184.031759] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [12187.523507] Key type lgssc unregistered [12187.884731] LNet: 411185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12187.898589] LNetError: 411185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12188.965787] LNet: Removed LNI 192.168.203.139@tcp [12189.829732] Key type .llcrypt unregistered [12189.832497] Key type ._llcrypt unregistered [12211.517090] Key type ._llcrypt registered [12211.522154] Key type .llcrypt registered [12211.619298] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [12211.938555] Key type .llcrypt unregistered [12211.943083] Key type ._llcrypt unregistered [12228.341895] Key type ._llcrypt registered [12228.343453] Key type .llcrypt registered [12228.410312] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [12229.081496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12229.090508] alg: No test for adler32 (adler32-zlib) [12230.075161] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12230.318640] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [12231.975426] Key type lgssc registered [12233.183967] Lustre: Echo OBD driver; http://www.lustre.org/ [12241.823727] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12241.902402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12243.498839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12248.568394] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12253.669241] LustreError: 413390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12253.733532] LustreError: 413390:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12258.029809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12258.340147] LustreError: 413391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12258.381048] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12263.400126] LustreError: 413385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12263.423574] LustreError: 413385:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12265.663638] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12268.516652] LustreError: 413390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12268.540585] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12268.550774] LustreError: 413390:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12273.645866] LustreError: 414590:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12273.688331] LustreError: 414590:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12274.123414] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12281.343412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12281.741090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12286.617735] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12297.188963] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12297.189577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12297.235490] Lustre: Skipped 1 previous similar message [12297.236831] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12297.276434] Lustre: Skipped 1 previous similar message [12302.312328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12302.332741] Lustre: Skipped 1 previous similar message [12302.829044] Lustre: server umount lustre-OST0000 complete [12307.428033] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12307.437647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12307.450144] Lustre: Skipped 1 previous similar message [12307.450482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12307.453970] Lustre: Skipped 1 previous similar message [12312.544930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12312.553636] Lustre: Skipped 1 previous similar message [12312.775771] Lustre: server umount lustre-MDT0000 complete [12316.748623] LustreError: 413371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451772 with bad export cookie 14548292566309708032 [12316.760530] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12317.201972] Lustre: server umount lustre-MDT0001 complete [12327.408293] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [12330.334479] Key type lgssc unregistered [12330.628804] LNet: 416535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12330.635370] LNetError: 416535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12330.651181] LNet: Removed LNI 192.168.203.139@tcp [12331.698969] Key type .llcrypt unregistered [12331.705650] Key type ._llcrypt unregistered [12356.626664] Key type ._llcrypt registered [12356.630513] Key type .llcrypt registered [12356.751939] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [12358.111357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12358.134717] alg: No test for adler32 (adler32-zlib) [12359.209431] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12359.468459] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [12361.167468] Key type lgssc registered [12362.346367] Lustre: Echo OBD driver; http://www.lustre.org/ [12371.145971] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12371.185845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12386.783849] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12391.903341] LustreError: 417921:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.139@tcp: failed processing log, type 4: rc = -110 [12418.591223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12425.038539] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12432.381088] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12439.712497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12440.304592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12440.374279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12445.671327] LustreError: 417947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12445.706030] LustreError: 417947:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12446.237478] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12450.805282] LustreError: 417946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12453.888493] LustreError: 418959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12453.935895] LustreError: 418959:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12454.923893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12455.432384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12460.328800] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12469.112733] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12476.278765] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12486.119232] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12486.137140] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12486.143783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12486.150282] Lustre: Skipped 1 previous similar message [12486.162062] Lustre: Skipped 1 previous similar message [12487.443237] Lustre: server umount lustre-OST0000 complete [12496.355457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12496.359827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12496.372641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12496.392178] Lustre: Skipped 1 previous similar message [12496.415032] Lustre: Skipped 1 previous similar message [12498.811928] Lustre: server umount lustre-MDT0000 complete [12501.473915] LustreError: 418963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12501.493135] LustreError: 418963:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12502.840430] LustreError: 417927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451958 with bad export cookie 9061056730895180655 [12502.846027] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12502.852314] LustreError: 417927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12503.119922] Lustre: server umount lustre-MDT0001 complete [12513.333940] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [12516.105191] Key type lgssc unregistered [12516.444686] LNet: 421492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12516.455107] LNetError: 421492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12516.498098] LNet: Removed LNI 192.168.203.139@tcp [12517.436566] Key type .llcrypt unregistered [12517.439499] Key type ._llcrypt unregistered [12541.904516] Key type ._llcrypt registered [12541.906848] Key type .llcrypt registered [12541.970934] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [12543.143723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12543.195530] alg: No test for adler32 (adler32-zlib) [12544.383335] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12544.648195] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [12546.263352] Key type lgssc registered [12547.359484] Lustre: Echo OBD driver; http://www.lustre.org/ [12556.023582] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12556.038405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12557.740592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12563.500715] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12568.036154] LustreError: 422908:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12568.063941] LustreError: 422908:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12573.153940] LustreError: 422907:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12573.190740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12573.611034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12578.826496] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12587.523669] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12593.540708] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12600.545549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12600.793823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12602.872649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12606.383861] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12615.799843] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12623.346122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12623.351089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12623.365729] Lustre: Skipped 1 previous similar message [12623.404051] Lustre: Skipped 2 previous similar messages [12623.762928] Lustre: server umount lustre-MDT0000 complete [12627.405276] LustreError: 422887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.39@tcp arrived at 1788452083 with bad export cookie 17838522894952290670 [12627.437339] LustreError: 422887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12628.471635] LustreError: 422907:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12628.499274] LustreError: 422907:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12629.642317] LustreError: 422888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452085 with bad export cookie 17838522894952289781 [12629.658935] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12629.667103] LustreError: 422888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12633.568412] LustreError: 422908:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12633.572283] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12633.593675] LustreError: 422908:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12633.606594] Lustre: Skipped 1 previous similar message [12633.606985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12633.614907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12633.651405] Lustre: Skipped 1 previous similar message [12635.915288] Lustre: server umount lustre-OST0000 complete [12639.715958] LustreError: 422908:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12639.751314] LustreError: 422908:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12643.708676] Lustre: server umount lustre-MDT0001 complete [12655.136413] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [12657.783176] Key type lgssc unregistered [12658.073706] LNet: 426591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12658.087447] LNetError: 426591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12658.108178] LNet: Removed LNI 192.168.203.139@tcp [12658.823970] Key type .llcrypt unregistered [12658.827425] Key type ._llcrypt unregistered [12683.670837] Key type ._llcrypt registered [12683.674288] Key type .llcrypt registered [12683.845770] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [12684.685095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12684.757972] alg: No test for adler32 (adler32-zlib) [12685.780772] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12686.028729] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [12687.743172] Key type lgssc registered [12688.873514] Lustre: Echo OBD driver; http://www.lustre.org/ [12696.348975] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12696.386147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12697.786671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12702.778344] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12707.307381] LustreError: 428008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12707.338786] LustreError: 428008:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12712.420799] LustreError: 428007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12712.840767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12713.576586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12720.026848] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12727.850521] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12734.234543] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12742.347807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12742.697956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12743.726732] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12748.302403] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12756.032537] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12763.236135] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.39@tcp (stopping) [12764.135063] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12764.153953] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12768.361616] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.39@tcp (stopping) [12768.381894] Lustre: Skipped 1 previous similar message [12773.479635] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.39@tcp (stopping) [12773.488432] Lustre: Skipped 2 previous similar messages [12774.367531] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12774.514753] Lustre: server umount lustre-MDT0001 complete [12779.496677] LustreError: 428736:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12779.524628] LustreError: 428736:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12784.610220] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12784.617393] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12784.627348] LustreError: 428007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12784.636493] Lustre: Skipped 1 previous similar message [12784.644550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12784.661759] Lustre: Skipped 2 previous similar messages [12786.855807] Lustre: server umount lustre-OST0000 complete [12789.731115] LustreError: 428736:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12790.909159] Lustre: server umount lustre-MDT0000 complete [12804.734503] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [12807.411656] Key type lgssc unregistered [12807.748400] LNet: 431693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12807.758804] LNetError: 431693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12807.779481] LNet: Removed LNI 192.168.203.139@tcp [12808.681226] Key type .llcrypt unregistered [12808.682853] Key type ._llcrypt unregistered [12833.258415] Key type ._llcrypt registered [12833.260215] Key type .llcrypt registered [12833.411901] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [12850.599411] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [12851.355622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12851.363987] alg: No test for adler32 (adler32-zlib) [12852.601689] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12852.850937] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [12854.519204] Key type lgssc registered [12855.804347] Lustre: Echo OBD driver; http://www.lustre.org/ [12898.939437] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [12916.699759] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12916.718595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12917.964593] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12917.998623] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12918.138176] Lustre: lustre-MDT0000: new disk, initializing [12918.238932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12918.261263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12922.837688] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12938.443246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12938.766593] Lustre: lustre-OST0000: new disk, initializing [12938.773956] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12938.785931] Lustre: Skipped 1 previous similar message [12938.804425] Lustre: 436724:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12938.877145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12945.213393] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12948.066854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12948.077896] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12948.129571] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12955.631855] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12972.455585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12972.610078] Lustre: 435709:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12972.720343] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12972.837406] Lustre: lustre-MDT0001: new disk, initializing [12973.000434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12973.044104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12973.055330] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12977.819129] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12990.216753] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12990.486914] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12996.053396] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12996.319895] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13003.901176] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.39@tcp (stopping) [13006.305372] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13006.314234] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13006.326901] Lustre: Skipped 1 previous similar message [13008.246537] Lustre: server umount lustre-MDT0001 complete [13014.108829] LustreError: 435712:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13014.144962] LustreError: 435712:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13016.551690] LustreError: 435718:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13016.571370] LustreError: 435718:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13021.669809] LustreError: 435712:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13021.689759] LustreError: 435712:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13026.788468] LustreError: 435712:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13026.830883] LustreError: 435712:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13029.856733] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13029.876775] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13029.898213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13029.905481] Lustre: Skipped 1 previous similar message [13031.908144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13031.909944] LustreError: 435717:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13034.561777] Lustre: server umount lustre-OST0000 complete [13045.601073] Lustre: server umount lustre-MDT0000 complete [13060.909170] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [13063.884456] Key type lgssc unregistered [13064.263742] LNet: 440478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13064.274190] LNetError: 440478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13064.289752] LNet: Removed LNI 192.168.203.139@tcp [13065.185967] Key type .llcrypt unregistered [13065.199400] Key type ._llcrypt unregistered [13110.268897] Key type ._llcrypt registered [13110.270991] Key type .llcrypt registered [13110.388691] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [13110.673300] Key type .llcrypt unregistered [13110.675182] Key type ._llcrypt unregistered [13128.308807] Key type ._llcrypt registered [13128.313622] Key type .llcrypt registered [13128.479616] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [13130.169654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13130.225170] alg: No test for adler32 (adler32-zlib) [13131.205146] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13131.410387] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [13133.055786] Key type lgssc registered [13134.473548] Lustre: Echo OBD driver; http://www.lustre.org/ [13165.288578] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13165.333094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13166.723044] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13166.748959] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13166.814925] Lustre: lustre-MDT0000: new disk, initializing [13166.897131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13166.919205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13171.196254] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [13181.575134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13181.748269] Lustre: 443850:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13181.811812] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13181.818122] Lustre: Skipped 1 previous similar message [13181.887370] Lustre: lustre-MDT0001: new disk, initializing [13181.981873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13182.022657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13182.040600] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13187.083736] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [13196.367948] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13202.118725] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13209.254593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13209.609743] Lustre: lustre-OST0000: new disk, initializing [13209.613997] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13209.629022] Lustre: 445819:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13209.711831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13216.374647] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [13217.876362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13217.888269] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13218.017683] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13225.544287] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13233.123345] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13233.136530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13233.153164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13238.597265] Lustre: server umount lustre-MDT0000 complete [13242.334220] LustreError: 443843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452698 with bad export cookie 17221422237542108060 [13242.356382] LustreError: 443843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13242.363403] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13242.736735] Lustre: server umount lustre-MDT0001 complete [13256.604181] Lustre: server umount lustre-OST0000 complete [13267.392237] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:25:21 (1788452721) [13275.025354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13275.423385] LustreError: 447848:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13275.438922] LustreError: 447848:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13275.490214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13280.105132] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [13280.738119] LustreError: 447849:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13285.862140] LustreError: 447848:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13288.727317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13289.172464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13293.420936] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [13301.378248] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13307.279488] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13327.734249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13327.930051] Lustre: Found index 0 for lustre-OST0000, updating log [13328.276856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13334.811780] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [13343.903653] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13346.673195] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13354.982919] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13354.991719] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13355.002582] Lustre: Skipped 2 previous similar messages [13355.009654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13355.014608] Lustre: Skipped 2 previous similar messages [13356.067176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13356.075719] Lustre: Skipped 1 previous similar message [13359.974980] Lustre: server umount lustre-OST0000 complete [13376.482593] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13376.491387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13376.519564] Lustre: Skipped 1 previous similar message [13376.528277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13376.534788] Lustre: Skipped 1 previous similar message [13382.526037] Lustre: server umount lustre-MDT0000 complete [13382.626705] LustreError: 447849:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13382.660103] LustreError: 447849:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13386.900708] LustreError: 447829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452842 with bad export cookie 17221422237542109453 [13386.915822] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13387.336290] Lustre: server umount lustre-MDT0001 complete [13394.591772] LNet: 1 local NIs in recovery (showing 1): 192.168.203.139@tcp [13397.269986] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:27:31 (1788452851) [14025.803300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14026.396710] LustreError: 454261:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14026.432361] LustreError: 454261:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14026.519595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14031.520055] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [14031.854729] LustreError: 454262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14036.961774] LustreError: 454261:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14040.786127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14041.195700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14045.778644] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [14055.278763] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14062.307292] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14083.505607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14083.720150] Lustre: Found index 0 for lustre-OST0000, updating log [14084.007204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14086.053455] LNet: 1 local NIs in recovery (showing 1): 192.168.203.139@tcp [14091.341966] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [14099.418082] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14102.500079] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14102.523194] Lustre: Skipped 1 previous similar message [14102.537323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14102.548229] Lustre: Skipped 2 previous similar messages [14107.842385] Lustre: server umount lustre-OST0000 complete [14112.738944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14112.757638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14112.783262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14113.828654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14113.838757] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14115.939083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14115.950312] Lustre: Skipped 4 previous similar messages [14118.117536] Lustre: server umount lustre-MDT0000 complete [14119.011491] LustreError: 454992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14119.040794] LustreError: 454992:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14122.360949] LustreError: 454243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453578 with bad export cookie 17221422237542110538 [14122.363841] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14122.377261] LustreError: 454243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [14122.861919] Lustre: server umount lustre-MDT0001 complete [14146.463794] LNet: 1 local NIs in recovery (showing 1): 192.168.203.139@tcp [14154.444616] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [14165.313528] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [14207.645758] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:41:01 (1788453661) [14228.427989] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [14232.021342] Key type lgssc unregistered [14232.350050] LNet: 460875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14232.361363] LNetError: 460875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14346.089692] LNet: Removed LNI 192.168.203.139@tcp [14347.051190] Key type .llcrypt unregistered [14347.053065] Key type ._llcrypt unregistered [14363.794115] Key type ._llcrypt registered [14363.799093] Key type .llcrypt registered [14363.908519] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [14364.929325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14364.938073] alg: No test for adler32 (adler32-zlib) [14366.063290] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [14366.395347] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [14368.031180] Key type lgssc registered [14369.359076] Lustre: Echo OBD driver; http://www.lustre.org/ [14375.006543] LNet: Added LNI 192.168.203.139@tcp100 [8/256/0/180] [14376.129784] LNet: Added LNI 192.168.203.139@tcp101 [8/256/0/180] [14377.259161] LNet: Added LNI 192.168.203.139@tcp102 [8/256/0/180] [14378.200440] LNet: Added LNI 192.168.203.139@tcp103 [8/256/0/180] [14379.446550] LNet: Added LNI 192.168.203.139@tcp104 [8/256/0/180] [14380.603648] LNet: Added LNI 192.168.203.139@tcp105 [8/256/0/180] [14381.761120] LNet: Added LNI 192.168.203.139@tcp106 [8/256/0/180] [14382.715270] LNet: Added LNI 192.168.203.139@tcp107 [8/256/0/180] [14383.823104] LNet: Added LNI 192.168.203.139@tcp108 [8/256/0/180] [14384.966477] LNet: Added LNI 192.168.203.139@tcp109 [8/256/0/180] [14385.945640] LNet: Added LNI 192.168.203.139@tcp110 [8/256/0/180] [14387.004051] LNet: Added LNI 192.168.203.139@tcp111 [8/256/0/180] [14388.132780] LNet: Added LNI 192.168.203.139@tcp112 [8/256/0/180] [14389.385209] LNet: Added LNI 192.168.203.139@tcp113 [8/256/0/180] [14390.738605] LNet: Added LNI 192.168.203.139@tcp114 [8/256/0/180] [14391.953639] LNet: Added LNI 192.168.203.139@tcp115 [8/256/0/180] [14393.037254] LNet: Added LNI 192.168.203.139@tcp116 [8/256/0/180] [14394.026364] LNet: Added LNI 192.168.203.139@tcp117 [8/256/0/180] [14395.193571] LNet: Added LNI 192.168.203.139@tcp118 [8/256/0/180] [14396.829337] LNet: Added LNI 192.168.203.139@tcp119 [8/256/0/180] [14397.972301] LNet: Added LNI 192.168.203.139@tcp120 [8/256/0/180] [14398.984480] LNet: Added LNI 192.168.203.139@tcp121 [8/256/0/180] [14400.200387] LNet: Added LNI 192.168.203.139@tcp122 [8/256/0/180] [14401.337892] LNet: Added LNI 192.168.203.139@tcp123 [8/256/0/180] [14402.824330] LNet: Added LNI 192.168.203.139@tcp124 [8/256/0/180] [14404.499988] LNet: Added LNI 192.168.203.139@tcp125 [8/256/0/180] [14406.266389] LNet: Added LNI 192.168.203.139@tcp126 [8/256/0/180] [14407.875386] LNet: Added LNI 192.168.203.139@tcp127 [8/256/0/180] [14409.173513] LNet: Added LNI 192.168.203.139@tcp128 [8/256/0/180] [14410.496322] LNet: Added LNI 192.168.203.139@tcp129 [8/256/0/180] [14411.781383] LNet: Added LNI 192.168.203.139@tcp130 [8/256/0/180] [14412.487768] LNet: Added LNI 192.168.203.139@tcp131 [8/256/0/180] [14413.497435] LNet: Added LNI 192.168.203.139@tcp132 [8/256/0/180] [14414.437209] LNet: Added LNI 192.168.203.139@tcp133 [8/256/0/180] [14415.773237] LNet: Added LNI 192.168.203.139@tcp134 [8/256/0/180] [14416.661895] LNet: Added LNI 192.168.203.139@tcp135 [8/256/0/180] [14424.593500] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14424.611121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14425.720215] Lustre: 463904:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.139@tcp: can't add NID 192.168.203.139@tcp130: rc = -75 [14425.843623] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14425.860927] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14425.923653] Lustre: lustre-MDT0000: new disk, initializing [14425.977225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14425.991333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14429.521251] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [14438.817607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14439.028804] Lustre: 463922:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14439.107673] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14439.112176] Lustre: Skipped 1 previous similar message [14439.279843] Lustre: lustre-MDT0001: new disk, initializing [14439.324853] Lustre: 464715:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.139@tcp: can't add NID 192.168.203.139@tcp130: rc = -75 [14439.334954] Lustre: 464715:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14439.432066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14439.460910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14439.479504] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14443.354703] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [14448.693145] Lustre: 465201:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.139@tcp: can't add NID 192.168.203.139@tcp130: rc = -75 [14448.699654] Lustre: 465201:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14451.906789] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14459.984221] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14468.683195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14469.172048] Lustre: lustre-OST0000: new disk, initializing [14469.179629] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14469.192382] Lustre: 465889:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14469.287986] Lustre: 465894:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.139@tcp: can't add NID 192.168.203.139@tcp130: rc = -75 [14469.292385] Lustre: 465894:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14469.324302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14475.097883] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [14478.881516] Lustre: 466406:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.139@tcp: can't add NID 192.168.203.139@tcp130: rc = -75 [14478.890227] Lustre: 466406:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14478.949456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14478.962764] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14479.064391] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14483.131823] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14494.701859] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14494.708460] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14494.720638] LustreError: Skipped 1 previous similar message [14494.740890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14494.776785] Lustre: Skipped 1 previous similar message [14496.072785] Lustre: server umount lustre-OST0000 complete [14504.929431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14504.948254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14504.952308] Lustre: Skipped 2 previous similar messages [14507.522639] Lustre: server umount lustre-MDT0000 complete [14511.795253] LustreError: 463912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453967 with bad export cookie 14615023836107553963 [14511.813458] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14512.446517] Lustre: server umount lustre-MDT0001 complete [14524.126396] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [14526.772713] Key type lgssc unregistered [14527.200300] LNet: 467801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14527.217910] LNetError: 467801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14527.229504] LNet: Removed LNI 192.168.203.139@tcp [14528.288291] LNet: Removed LNI 192.168.203.139@tcp100 [14528.291661] LNetError: 467801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14528.303525] LNetError: 467801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14528.317593] LNet: Removed LNI 192.168.203.139@tcp101 [14528.324737] LNet: Removed LNI 192.168.203.139@tcp102 [14528.330782] LNet: Removed LNI 192.168.203.139@tcp103 [14528.337319] LNet: Removed LNI 192.168.203.139@tcp104 [14528.343961] LNet: Removed LNI 192.168.203.139@tcp105 [14528.350731] LNet: Removed LNI 192.168.203.139@tcp106 [14528.357742] LNet: Removed LNI 192.168.203.139@tcp107 [14528.365157] LNet: Removed LNI 192.168.203.139@tcp108 [14528.368472] LNet: Removed LNI 192.168.203.139@tcp109 [14528.378929] LNet: Removed LNI 192.168.203.139@tcp110 [14528.382640] LNet: Removed LNI 192.168.203.139@tcp111 [14528.388950] LNet: Removed LNI 192.168.203.139@tcp112 [14528.394367] LNet: Removed LNI 192.168.203.139@tcp113 [14528.401395] LNet: Removed LNI 192.168.203.139@tcp114 [14528.405408] LNet: Removed LNI 192.168.203.139@tcp115 [14528.409443] LNet: Removed LNI 192.168.203.139@tcp116 [14528.412775] LNet: Removed LNI 192.168.203.139@tcp117 [14528.416132] LNet: Removed LNI 192.168.203.139@tcp118 [14528.419490] LNet: Removed LNI 192.168.203.139@tcp119 [14528.422652] LNet: Removed LNI 192.168.203.139@tcp120 [14528.425941] LNet: Removed LNI 192.168.203.139@tcp121 [14528.429541] LNet: Removed LNI 192.168.203.139@tcp122 [14528.432892] LNet: Removed LNI 192.168.203.139@tcp123 [14528.444076] LNet: Removed LNI 192.168.203.139@tcp124 [14528.448965] LNet: Removed LNI 192.168.203.139@tcp125 [14528.453766] LNet: Removed LNI 192.168.203.139@tcp126 [14528.457072] LNet: Removed LNI 192.168.203.139@tcp127 [14528.461915] LNet: Removed LNI 192.168.203.139@tcp128 [14528.466773] LNet: Removed LNI 192.168.203.139@tcp129 [14528.473066] LNet: Removed LNI 192.168.203.139@tcp130 [14528.480378] LNet: Removed LNI 192.168.203.139@tcp131 [14528.486876] LNet: Removed LNI 192.168.203.139@tcp132 [14528.493865] LNet: Removed LNI 192.168.203.139@tcp133 [14528.502780] LNet: Removed LNI 192.168.203.139@tcp134 [14528.524172] LNet: Removed LNI 192.168.203.139@tcp135 [14529.590222] Key type .llcrypt unregistered [14529.594327] Key type ._llcrypt unregistered [14553.140082] Key type ._llcrypt registered [14553.142448] Key type .llcrypt registered [14553.249717] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [14566.112316] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [14567.323069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14567.335158] alg: No test for adler32 (adler32-zlib) [14568.382840] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [14568.611981] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [14570.303164] Key type lgssc registered [14571.482496] Lustre: Echo OBD driver; http://www.lustre.org/ [14611.123990] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 12:47:45 (1788454065) [14629.874579] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [14632.831289] Key type lgssc unregistered [14633.080494] LNet: 471862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14633.091318] LNetError: 471862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14633.109165] LNet: Removed LNI 192.168.203.139@tcp [14633.925350] Key type .llcrypt unregistered [14633.927103] Key type ._llcrypt unregistered [14654.308869] Key type ._llcrypt registered [14654.314646] Key type .llcrypt registered [14654.551236] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [14656.089173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14656.139213] alg: No test for adler32 (adler32-zlib) [14657.289246] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [14657.538297] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [14659.215152] Key type lgssc registered [14660.188841] Lustre: Echo OBD driver; http://www.lustre.org/ [14669.567808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14669.602924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14670.752225] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14670.771532] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14670.815853] Lustre: lustre-MDT0000: new disk, initializing [14670.860520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14670.883879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14674.746213] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [14686.296815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14686.436298] Lustre: 473073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14686.506750] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14686.511266] Lustre: Skipped 1 previous similar message [14686.570481] Lustre: lustre-MDT0001: new disk, initializing [14686.667585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14686.702149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14686.709185] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14691.243457] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [14702.779661] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14709.991525] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14718.711151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14719.252145] Lustre: lustre-OST0000: new disk, initializing [14719.262769] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14719.296837] Lustre: 475040:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14719.487237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14726.238293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14726.262780] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14726.421311] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14726.479756] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [14735.335467] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14739.137193] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing lnet_if_list [14744.060898] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14744.089824] LNet: Added LNI 192.168.203.139@tcp10 [8/256/0/180] [14749.735234] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14749.780819] LNet: Added LNI 192.168.203.139@tcp11 [8/256/0/180] [14754.485307] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14754.509432] LNet: Added LNI 192.168.203.139@tcp12 [8/256/0/180] [14758.930276] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14758.973317] LNet: Added LNI 192.168.203.139@tcp13 [8/256/0/180] [14763.658312] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14763.704413] LNet: Added LNI 192.168.203.139@tcp14 [8/256/0/180] [14769.034975] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14769.067582] LNet: Added LNI 192.168.203.139@tcp15 [8/256/0/180] [14774.528073] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14774.597857] LNet: Added LNI 192.168.203.139@tcp16 [8/256/0/180] [14779.739199] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14779.766256] LNet: Added LNI 192.168.203.139@tcp17 [8/256/0/180] [14784.668836] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14784.718794] LNet: Added LNI 192.168.203.139@tcp18 [8/256/0/180] [14790.618353] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14790.684768] LNet: Added LNI 192.168.203.139@tcp19 [8/256/0/180] [14796.031345] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14796.058208] LNet: Added LNI 192.168.203.139@tcp20 [8/256/0/180] [14803.253536] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14803.304149] LNet: Added LNI 192.168.203.139@tcp21 [8/256/0/180] [14810.715157] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14810.738677] LNet: Added LNI 192.168.203.139@tcp22 [8/256/0/180] [14817.192119] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14817.235523] LNet: Added LNI 192.168.203.139@tcp23 [8/256/0/180] [14823.270610] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14823.330494] LNet: Added LNI 192.168.203.139@tcp24 [8/256/0/180] [14828.905427] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14828.930079] LNet: Added LNI 192.168.203.139@tcp25 [8/256/0/180] [14836.035681] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14836.059943] LNet: Added LNI 192.168.203.139@tcp26 [8/256/0/180] [14842.283450] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14842.301814] LNet: Added LNI 192.168.203.139@tcp27 [8/256/0/180] [14849.257525] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14849.301852] LNet: Added LNI 192.168.203.139@tcp28 [8/256/0/180] [14856.715595] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14856.732392] LNet: Added LNI 192.168.203.139@tcp29 [8/256/0/180] [14864.812545] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14864.860448] LNet: Added LNI 192.168.203.139@tcp30 [8/256/0/180] [14872.819678] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14872.861059] LNet: Added LNI 192.168.203.139@tcp31 [8/256/0/180] [14879.847589] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14879.880188] LNet: Added LNI 192.168.203.139@tcp32 [8/256/0/180] [14886.770931] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14886.808896] LNet: Added LNI 192.168.203.139@tcp33 [8/256/0/180] [14893.063463] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14893.107841] LNet: Added LNI 192.168.203.139@tcp34 [8/256/0/180] [14899.790444] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14899.820625] LNet: Added LNI 192.168.203.139@tcp35 [8/256/0/180] [14907.439456] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14907.482308] LNet: Added LNI 192.168.203.139@tcp36 [8/256/0/180] [14915.657696] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14915.700430] LNet: Added LNI 192.168.203.139@tcp37 [8/256/0/180] [14923.463924] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14923.493936] LNet: Added LNI 192.168.203.139@tcp38 [8/256/0/180] [14930.307468] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14930.377328] LNet: Added LNI 192.168.203.139@tcp39 [8/256/0/180] [14937.931235] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14937.963996] LNet: Added LNI 192.168.203.139@tcp40 [8/256/0/180] [14945.852193] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14945.908170] LNet: Added LNI 192.168.203.139@tcp41 [8/256/0/180] [14956.247742] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14956.269961] LNet: Added LNI 192.168.203.139@tcp42 [8/256/0/180] [14966.290600] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14966.378162] LNet: Added LNI 192.168.203.139@tcp43 [8/256/0/180] [14974.063685] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14974.098994] LNet: Added LNI 192.168.203.139@tcp44 [8/256/0/180] [14982.427338] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14982.483425] LNet: Added LNI 192.168.203.139@tcp45 [8/256/0/180] [14991.639590] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14991.703529] LNet: Added LNI 192.168.203.139@tcp46 [8/256/0/180] [15000.858984] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15000.903051] LNet: Added LNI 192.168.203.139@tcp47 [8/256/0/180] [15011.137887] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15011.211302] LNet: Added LNI 192.168.203.139@tcp48 [8/256/0/180] [15020.840018] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15020.860383] LNet: Added LNI 192.168.203.139@tcp49 [8/256/0/180] [15029.406271] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15029.432328] LNet: Added LNI 192.168.203.139@tcp50 [8/256/0/180] [15049.184974] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15049.197960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15049.215277] Lustre: Skipped 1 previous similar message [15053.369147] Lustre: server umount lustre-MDT0000 complete [15057.940115] LustreError: 473877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454513 with bad export cookie 47393286336575679 [15057.940520] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15057.968915] LustreError: 473877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15058.505412] Lustre: server umount lustre-MDT0001 complete [15075.616128] Lustre: 472344:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788454515/real 1788454515] req@000000004ea77a48 x1875330060167296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788454531 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15075.673368] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15075.696189] Lustre: Skipped 1 previous similar message [15078.880077] Lustre: 472343:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788454518/real 1788454518] req@00000000f68bb787 x1875330060167424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788454534 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15079.214623] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [15080.927311] Lustre: 472342:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788454520/real 1788454520] req@0000000086e6249f x1875330060167552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788454536 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15083.999827] Lustre: 472345:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788454523/real 1788454523] req@000000008af50f78 x1875330060167808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788454539 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15088.161887] Lustre: 472345:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788454528/real 1788454528] req@000000007f9f7314 x1875330060167936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788454544 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15088.995863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15102.370229] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xa85ff15cd482a5 [15102.399103] Lustre: MGC192.168.203.139@tcp: Connection restored to 0@lo (at 0@lo) [15102.749634] LustreError: 475034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15102.763962] LustreError: 475034:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [15102.817180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15104.868873] LustreError: 475035:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15104.879514] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15105.890048] LustreError: 475034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15107.335400] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [15111.150104] LustreError: 475035:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15116.257440] LustreError: 475034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15117.195695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15117.596186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15120.688408] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15122.303699] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [15130.341596] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15136.778373] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15161.825075] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15161.827752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15161.833349] Lustre: Skipped 1 previous similar message [15161.837152] Lustre: Skipped 2 previous similar messages [15166.570925] Lustre: server umount lustre-OST0000 complete [15171.181494] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15171.192159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15171.207419] Lustre: Skipped 1 previous similar message [15171.220187] Lustre: Skipped 1 previous similar message [15176.672470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15177.620523] Lustre: server umount lustre-MDT0000 complete [15181.797974] LustreError: 501146:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15181.821721] LustreError: 501146:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15181.910144] LustreError: 473064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454637 with bad export cookie 47393286336578213 [15181.918949] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15181.927211] LustreError: 473064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15182.287122] Lustre: server umount lustre-MDT0001 complete [15194.631909] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [15197.384428] Key type lgssc unregistered [15197.818334] LNet: 502087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15197.826163] LNetError: 502087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15197.847915] LNet: Removed LNI 192.168.203.139@tcp [15197.851898] LNet: Removed LNI 192.168.203.139@tcp10 [15197.857384] LNet: Removed LNI 192.168.203.139@tcp11 [15197.863714] LNet: Removed LNI 192.168.203.139@tcp12 [15197.867787] LNet: Removed LNI 192.168.203.139@tcp13 [15197.874125] LNet: Removed LNI 192.168.203.139@tcp14 [15197.876724] LNet: Removed LNI 192.168.203.139@tcp15 [15197.879358] LNet: Removed LNI 192.168.203.139@tcp16 [15197.882812] LNet: Removed LNI 192.168.203.139@tcp17 [15197.884811] LNet: Removed LNI 192.168.203.139@tcp18 [15197.886847] LNet: Removed LNI 192.168.203.139@tcp19 [15197.891418] LNet: Removed LNI 192.168.203.139@tcp20 [15197.894588] LNet: Removed LNI 192.168.203.139@tcp21 [15197.897746] LNet: Removed LNI 192.168.203.139@tcp22 [15197.900584] LNet: Removed LNI 192.168.203.139@tcp23 [15197.902914] LNet: Removed LNI 192.168.203.139@tcp24 [15197.905878] LNet: Removed LNI 192.168.203.139@tcp25 [15197.908785] LNet: Removed LNI 192.168.203.139@tcp26 [15197.918329] LNet: Removed LNI 192.168.203.139@tcp27 [15197.924817] LNet: Removed LNI 192.168.203.139@tcp28 [15197.928789] LNet: Removed LNI 192.168.203.139@tcp29 [15197.935970] LNet: Removed LNI 192.168.203.139@tcp30 [15197.939769] LNet: Removed LNI 192.168.203.139@tcp31 [15197.941799] LNet: Removed LNI 192.168.203.139@tcp32 [15197.949259] LNet: Removed LNI 192.168.203.139@tcp33 [15197.953561] LNet: Removed LNI 192.168.203.139@tcp34 [15197.958309] LNet: Removed LNI 192.168.203.139@tcp35 [15197.963196] LNet: Removed LNI 192.168.203.139@tcp36 [15197.968310] LNet: Removed LNI 192.168.203.139@tcp37 [15197.973352] LNet: Removed LNI 192.168.203.139@tcp38 [15197.976276] LNet: Removed LNI 192.168.203.139@tcp39 [15197.978619] LNet: Removed LNI 192.168.203.139@tcp40 [15197.980951] LNet: Removed LNI 192.168.203.139@tcp41 [15197.983699] LNet: Removed LNI 192.168.203.139@tcp42 [15197.986180] LNet: Removed LNI 192.168.203.139@tcp43 [15197.989625] LNet: Removed LNI 192.168.203.139@tcp44 [15197.992796] LNet: Removed LNI 192.168.203.139@tcp45 [15197.995888] LNet: Removed LNI 192.168.203.139@tcp46 [15197.999975] LNet: Removed LNI 192.168.203.139@tcp47 [15198.002865] LNet: Removed LNI 192.168.203.139@tcp48 [15198.006397] LNet: Removed LNI 192.168.203.139@tcp49 [15198.017688] LNet: Removed LNI 192.168.203.139@tcp50 [15198.865084] Key type .llcrypt unregistered [15198.867964] Key type ._llcrypt unregistered [15221.451330] Key type ._llcrypt registered [15221.453452] Key type .llcrypt registered [15221.558627] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [15239.887884] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [15240.949265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15241.158376] alg: No test for adler32 (adler32-zlib) [15242.727884] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [15243.229847] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [15244.911430] Key type lgssc registered [15247.194912] Lustre: Echo OBD driver; http://www.lustre.org/ [15294.857811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15294.890993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15296.298526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15296.401322] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15296.572528] Lustre: lustre-MDT0000: new disk, initializing [15296.686755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15296.706840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15302.603964] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [15315.253656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15315.384437] Lustre: 505281:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15315.478188] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15315.496232] Lustre: Skipped 1 previous similar message [15315.641647] Lustre: lustre-MDT0001: new disk, initializing [15315.778427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15315.865387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15315.877246] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15320.976919] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [15331.292090] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15336.971933] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15346.962767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15347.227953] Lustre: *** cfs_fail_loc=90f, val=0*** [15347.343320] Lustre: lustre-OST0000: new disk, initializing [15347.352784] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15347.363870] Lustre: 507301:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15347.480853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15354.104685] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [15357.047250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15357.067284] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15357.178634] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15364.389487] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15372.774255] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15372.776431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15372.791590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15377.895976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15377.910284] Lustre: Skipped 2 previous similar messages [15378.833725] Lustre: server umount lustre-OST0000 complete [15387.617879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15387.643308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15387.662572] Lustre: Skipped 1 previous similar message [15387.675964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15389.985208] Lustre: server umount lustre-MDT0000 complete [15395.184250] LustreError: 505274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454851 with bad export cookie 7992729305292873802 [15395.211800] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15396.015827] Lustre: server umount lustre-MDT0001 complete [15411.542772] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [15415.509392] Key type lgssc unregistered [15415.891234] LNet: 509218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15415.909982] LNetError: 509218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15415.924870] LNet: Removed LNI 192.168.203.139@tcp [15417.168692] Key type .llcrypt unregistered [15417.172149] Key type ._llcrypt unregistered [15442.732744] Key type ._llcrypt registered [15442.738795] Key type .llcrypt registered [15442.966427] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [15445.146964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15445.264315] alg: No test for adler32 (adler32-zlib) [15446.585380] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [15447.198871] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [15449.127510] Key type lgssc registered [15450.444059] Lustre: Echo OBD driver; http://www.lustre.org/ [15463.050779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15463.082381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15464.639200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15469.930428] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [15474.159351] LustreError: 510486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15474.177459] LustreError: 510486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15479.265927] LustreError: 510487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15479.789975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15480.264521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15485.745523] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [15494.579848] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15501.782943] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15509.701310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15510.044749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15519.480420] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [15529.110656] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15541.729182] loop0: detected capacity change from 0 to 204800000 [15541.966558] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15541.982480] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15542.001922] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15548.856141] loop0: detected capacity change from 0 to 204800000 [15549.173679] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15549.202468] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15549.218153] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15555.034515] loop0: detected capacity change from 0 to 204800000 [15561.027286] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15561.251228] Lustre: fs15246-OST0000: new disk, initializing [15561.256497] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15561.264143] Lustre: 514080:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15561.322601] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15568.521378] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [15584.239919] loop1: detected capacity change from 0 to 204800000 [15592.423609] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15592.595780] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15592.694770] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15592.780125] Lustre: fs15246-MDT0000: new disk, initializing [15592.940688] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15593.010497] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15593.155258] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15594.209646] Lustre: 515203:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000006b365366 x1875330888421376/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788455066 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15599.221290] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [15603.309193] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15603.319881] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15603.366343] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15628.276136] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15628.289641] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15630.181097] Lustre: server umount fs15246-MDT0000 complete [15633.380640] LustreError: 510481:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15638.515357] LustreError: 510481:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15642.251884] Lustre: server umount fs15246-OST0000 complete [15658.477347] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15658.482422] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15658.486730] LustreError: Skipped 1 previous similar message [15658.491808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15658.506567] Lustre: Skipped 1 previous similar message [15658.538949] Lustre: Skipped 1 previous similar message [15662.919478] Lustre: server umount lustre-OST0000 complete [15669.229742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15669.243251] Lustre: Skipped 1 previous similar message [15669.247525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15673.609276] Lustre: server umount lustre-MDT0000 complete [15674.336422] LustreError: 513548:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15674.356598] LustreError: 513548:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15677.791420] LustreError: 510469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455133 with bad export cookie 11380820197608007774 [15677.795113] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15677.809334] LustreError: 510469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15678.403742] Lustre: server umount lustre-MDT0001 complete [15693.455020] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [15697.862034] Key type lgssc unregistered [15698.357457] LNet: 518304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15698.371519] LNetError: 518304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15698.398177] LNet: Removed LNI 192.168.203.139@tcp [15698.967758] Key type .llcrypt unregistered [15698.970170] Key type ._llcrypt unregistered [15723.280505] Key type ._llcrypt registered [15723.305311] Key type .llcrypt registered [15723.446952] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [15724.773342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15724.789361] alg: No test for adler32 (adler32-zlib) [15726.067133] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [15726.440637] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [15728.151180] Key type lgssc registered [15730.237066] Lustre: Echo OBD driver; http://www.lustre.org/ [15782.911919] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [15794.102532] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [15844.980289] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15845.039373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15846.443827] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15846.512328] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15846.586257] Lustre: lustre-MDT0000: new disk, initializing [15846.743052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15846.783263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15851.553628] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [15864.127290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15864.303662] Lustre: 522773:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15864.426969] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15864.437743] Lustre: Skipped 1 previous similar message [15864.560375] Lustre: lustre-MDT0001: new disk, initializing [15864.697558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15864.776316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15864.810012] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15870.258185] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [15883.176276] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15891.305444] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15899.652609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15899.914386] Lustre: lustre-OST0000: new disk, initializing [15899.917995] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15899.924830] Lustre: 524745:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15900.041046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15907.963328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15907.987431] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15908.187082] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [15908.219484] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15919.137921] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15924.147416] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15924.318166] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15929.635954] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15929.843747] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15933.926156] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15933.928843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15933.940799] LustreError: Skipped 1 previous similar message [15933.946319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15933.957833] Lustre: Skipped 1 previous similar message [15933.982862] Lustre: Skipped 1 previous similar message [15938.358873] Lustre: server umount lustre-OST0000 complete [15944.167059] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15944.172570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15944.193804] Lustre: Skipped 1 previous similar message [15944.224477] Lustre: Skipped 1 previous similar message [15949.310110] Lustre: server umount lustre-MDT0000 complete [15953.893967] LustreError: 522764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455409 with bad export cookie 11376987182785701567 [15953.899655] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15953.915095] LustreError: 522764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15954.342800] Lustre: server umount lustre-MDT0001 complete [15986.380261] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [16000.552813] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [16042.230450] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15785 sec ========= 13:11:36 (1788455496) [16043.984376] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:11:38 (1788455498) === [16048.093869] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:11:41 (1788455501) === [16094.018633] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [16097.855630] Key type lgssc unregistered [16098.229631] LNet: 531415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16098.245589] LNetError: 531415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16098.270088] LNet: Removed LNI 192.168.203.139@tcp [16099.555660] Key type .llcrypt unregistered [16099.571400] Key type ._llcrypt unregistered