[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 762211413 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003260] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008828] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011127] LSM: Security Framework initializing [ 0.012054] Yama: becoming mindful. [ 0.013036] SELinux: Initializing. [ 0.014081] *** VALIDATE selinux *** [ 0.027026] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.033404] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.034146] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.035106] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.036120] *** VALIDATE tmpfs *** [ 0.037579] *** VALIDATE proc *** [ 0.038338] *** VALIDATE cgroup *** [ 0.039011] *** VALIDATE cgroup2 *** [ 0.041081] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.042134] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.043011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.044027] Spectre V2 : User space: Vulnerable [ 0.045009] Speculative Store Bypass: Vulnerable [ 0.048304] debug: unmapping init [mem 0xffffffffb7459000-0xffffffffb7460fff] [ 0.051345] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.053030] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.054023] ... version: 2 [ 0.055015] ... bit width: 48 [ 0.056013] ... generic registers: 4 [ 0.057014] ... value mask: 0000ffffffffffff [ 0.058019] ... max period: 00007fffffffffff [ 0.059017] ... fixed-purpose events: 3 [ 0.060013] ... event mask: 000000070000000f [ 0.061459] rcu: Hierarchical SRCU implementation. [ 0.063710] smp: Bringing up secondary CPUs ... [ 0.064668] x86: Booting SMP configuration: [ 0.065020] .... node #0, CPUs: #1 #2 #3 [ 0.074219] smp: Brought up 1 node, 4 CPUs [ 0.076015] smpboot: Max logical packages: 1 [ 0.077015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.127062] node 0 deferred pages initialised in 48ms [ 0.130398] devtmpfs: initialized [ 0.132283] x86/mm: Memory block size: 128MB [ 0.133000] gcov: version magic: 0x41383552 [ 0.137325] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.139132] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.141348] pinctrl core: initialized pinctrl subsystem [ 0.143210] [ 0.143708] ************************************************************* [ 0.147012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.150009] ** ** [ 0.153012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.156011] ** ** [ 0.158009] ** This means that this kernel is built to expose internal ** [ 0.160009] ** IOMMU data structures, which may compromise security on ** [ 0.162011] ** your system. ** [ 0.164011] ** ** [ 0.166009] ** If you see this message and you are not debugging the ** [ 0.169011] ** kernel, report this immediately to your vendor! ** [ 0.171007] ** ** [ 0.173010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.176011] ************************************************************* [ 0.180145] NET: Registered protocol family 16 [ 0.183448] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.188078] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.193069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.200344] cpuidle: using governor menu [ 0.204070] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.227721] PCI: Using configuration type 1 for base access [ 0.230390] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.295359] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.296013] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.300161] cryptd: max_cpu_qlen set to 1000 [ 0.302111] ACPI: Added _OSI(Module Device) [ 0.303012] ACPI: Added _OSI(Processor Device) [ 0.305014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.307013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.314262] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.321365] ACPI: Interpreter enabled [ 0.323123] ACPI: PM: (supports S0 S3 S4 S5) [ 0.325014] ACPI: Using IOAPIC for interrupt routing [ 0.327095] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.331424] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.344035] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.347034] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.350018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.354098] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.360732] acpiphp: Slot [2] registered [ 0.363260] acpiphp: Slot [5] registered [ 0.365193] acpiphp: Slot [6] registered [ 0.367161] acpiphp: Slot [7] registered [ 0.369121] acpiphp: Slot [8] registered [ 0.370176] acpiphp: Slot [9] registered [ 0.373130] acpiphp: Slot [10] registered [ 0.374210] acpiphp: Slot [3] registered [ 0.376097] acpiphp: Slot [4] registered [ 0.377097] acpiphp: Slot [11] registered [ 0.378111] acpiphp: Slot [12] registered [ 0.380105] acpiphp: Slot [13] registered [ 0.382110] acpiphp: Slot [14] registered [ 0.383189] acpiphp: Slot [15] registered [ 0.385113] acpiphp: Slot [16] registered [ 0.387210] acpiphp: Slot [17] registered [ 0.389192] acpiphp: Slot [18] registered [ 0.391099] acpiphp: Slot [19] registered [ 0.392166] acpiphp: Slot [20] registered [ 0.393197] acpiphp: Slot [21] registered [ 0.394101] acpiphp: Slot [22] registered [ 0.395121] acpiphp: Slot [23] registered [ 0.397175] acpiphp: Slot [24] registered [ 0.399098] acpiphp: Slot [25] registered [ 0.400096] acpiphp: Slot [26] registered [ 0.402170] acpiphp: Slot [27] registered [ 0.404111] acpiphp: Slot [28] registered [ 0.405248] acpiphp: Slot [29] registered [ 0.407097] acpiphp: Slot [30] registered [ 0.409106] acpiphp: Slot [31] registered [ 0.411092] PCI host bridge to bus 0000:00 [ 0.412018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.415019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.418019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.421024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.423110] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.425024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.426000] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.428190] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.432554] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.442856] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.448556] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.450017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.452026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.455018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.460566] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.463954] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.465044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.468863] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.473013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.485017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.491013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.498000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.506014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.518016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.539017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.555566] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.570038] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.577060] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.597024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.615787] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.625015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.632015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.656020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.675240] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.692016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.701015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.736017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.752000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.767015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.785021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.824017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.842667] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.848110] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.855019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.871023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.877754] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.880359] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.882535] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.884257] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.886180] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.889075] iommu: Default domain type: Passthrough [ 0.891448] SCSI subsystem initialized [ 0.893138] ACPI: bus type USB registered [ 0.894110] usbcore: registered new interface driver usbfs [ 0.896080] usbcore: registered new interface driver hub [ 0.899093] usbcore: registered new device driver usb [ 0.900199] pps_core: LinuxPPS API ver. 1 registered [ 0.902011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.906058] PTP clock support registered [ 0.908187] EDAC MC: Ver: 3.0.0 [ 0.910085] PCI: Using ACPI for IRQ routing [ 0.912614] NetLabel: Initializing [ 0.914007] NetLabel: domain hash size = 128 [ 0.916010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.918124] NetLabel: unlabeled traffic allowed by default [ 0.920125] vgaarb: loaded [ 0.922254] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.924013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.930707] clocksource: Switched to clocksource kvm-clock [ 1.198529] VFS: Disk quotas dquot_6.6.0 [ 1.207881] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.221523] *** VALIDATE ramfs *** [ 1.228508] *** VALIDATE hugetlbfs *** [ 1.232899] pnp: PnP ACPI init [ 1.239404] pnp: PnP ACPI: found 6 devices [ 1.279158] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.286765] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.290175] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.293805] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.298552] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.303685] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.309645] NET: Registered protocol family 2 [ 1.315571] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.328159] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.334609] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.342830] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.350482] TCP: Hash tables configured (established 65536 bind 65536) [ 1.356631] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.362603] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.369376] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.376877] NET: Registered protocol family 1 [ 1.382981] RPC: Registered named UNIX socket transport module. [ 1.387386] RPC: Registered udp transport module. [ 1.391551] RPC: Registered tcp transport module. [ 1.398587] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.404396] NET: Registered protocol family 44 [ 1.407618] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.411770] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.415934] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.420395] PCI: CLS 0 bytes, default 64 [ 1.423490] Unpacking initramfs... [ 4.332315] debug: unmapping init [mem 0xffff9871fcc54000-0xffff9871fffbffff] [ 4.344751] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.351796] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.357834] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.392045] Initialise system trusted keyrings [ 5.394593] Key type blacklist registered [ 5.397062] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.407651] zbud: loaded [ 5.410904] *** VALIDATE nfs *** [ 5.412335] *** VALIDATE nfs4 *** [ 5.414399] pstore: using deflate compression [ 5.419068] Platform Keyring initialized [ 5.612549] NET: Registered protocol family 38 [ 5.618394] Key type asymmetric registered [ 5.620497] Asymmetric key parser 'x509' registered [ 5.624267] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.632247] io scheduler mq-deadline registered [ 5.633798] io scheduler kyber registered [ 5.636358] io scheduler bfq registered [ 5.639347] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.644235] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.649665] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.653781] ACPI: Power Button [PWRF] [ 5.663498] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.675452] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.698956] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.709119] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.729522] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.760587] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.791392] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.797113] Non-volatile memory driver v1.3 [ 5.799216] Linux agpgart interface v0.103 [ 5.833712] virtio_blk virtio1: [vda] 145888 512-byte logical blocks (74.7 MB/71.2 MiB) [ 5.839258] vda: detected capacity change from 0 to 74694656 [ 5.855281] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.859208] vdb: detected capacity change from 0 to 1073741824 [ 5.880561] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.884282] vdc: detected capacity change from 0 to 2621440000 [ 5.902081] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.905911] vdd: detected capacity change from 0 to 2621440000 [ 5.922897] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.926604] vde: detected capacity change from 0 to 4294967296 [ 5.960647] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.966335] vdf: detected capacity change from 0 to 4294967296 [ 5.985827] libphy: Fixed MDIO Bus: probed [ 5.992474] usbcore: registered new interface driver usbserial_generic [ 5.995753] usbserial: USB Serial support registered for generic [ 6.001062] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.008608] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.013471] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.019494] mousedev: PS/2 mouse device common for all mice [ 6.024522] rtc_cmos 00:05: RTC can wake from S4 [ 6.029590] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.033139] rtc_cmos 00:05: registered as rtc0 [ 6.033329] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.033372] intel_pstate: CPU model not supported [ 6.043638] hid: raw HID events driver (C) Jiri Kosina [ 6.071226] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.074128] usbcore: registered new interface driver usbhid [ 6.086979] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.092891] usbhid: USB HID core driver [ 6.093136] drop_monitor: Initializing network drop monitor service [ 6.093438] Initializing XFRM netlink socket [ 6.094649] NET: Registered protocol family 10 [ 6.114896] Segment Routing with IPv6 [ 6.116395] NET: Registered protocol family 17 [ 6.118637] mpls_gso: MPLS GSO support [ 6.125106] RAS: Correctable Errors collector initialized. [ 6.127215] AVX version of gcm_enc/dec engaged. [ 6.128858] AES CTR mode by8 optimization enabled [ 6.328876] sched_clock: Marking stable (6328728701, 0)->(7515244118, -1186515417) [ 6.335362] registered taskstats version 1 [ 6.338889] Loading compiled-in X.509 certificates [ 6.342099] zswap: loaded using pool lzo/zbud [ 6.391983] Key type big_key registered [ 6.420137] Key type encrypted registered [ 6.422487] ima: No TPM chip found, activating TPM-bypass! [ 6.429950] ima: Allocated hash algorithm: sha1 [ 6.434394] ima: No architecture policies found [ 6.439378] evm: Initialising EVM extended attributes: [ 6.441987] evm: security.selinux [ 6.444097] evm: security.ima [ 6.446104] evm: security.capability [ 6.448789] evm: HMAC attrs: 0x1 [ 6.453313] rtc_cmos 00:05: setting system clock to 2026-08-07 00:25:07 UTC (1786062307) [ 6.464699] debug: unmapping init [mem 0xffffffffb8403000-0xffffffffb85fffff] [ 6.468923] debug: unmapping init [mem 0xffffffffb7182000-0xffffffffb7458fff] [ 6.484453] Write protecting the kernel read-only data: 28672k [ 6.495072] debug: unmapping init [mem 0xffffffffb5803000-0xffffffffb59fffff] [ 6.499366] debug: unmapping init [mem 0xffffffffb6114000-0xffffffffb61fffff] [ 6.619372] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.638457] systemd[1]: Detected virtualization kvm. [ 6.642648] systemd[1]: Detected architecture x86-64. [ 6.648977] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.687138] systemd[1]: No hostname configured. [ 6.689242] systemd[1]: Set hostname to . [ 6.691671] random: systemd: uninitialized urandom read (16 bytes read) [ 6.694763] systemd[1]: Initializing machine ID from random generator. [ 6.855951] random: ln: uninitialized urandom read (6 bytes read) [ 7.180768] random: systemd: uninitialized urandom read (16 bytes read) [ 7.186472] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 7.197820] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 7.207623] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.964490] device-mapper: uevent: version 1.0.3 [ 8.967295] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ 10.878288] virtio_net virtio0 ens2: renamed from eth0 [ 10.908695] random: fast init done [ 11.050552] scsi host0: ata_piix [ 11.108240] scsi host1: ata_piix [ 11.120287] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 11.144107] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 18.199344] random: crng init done [ 18.201858] random: 7 urandom warning(s) missed due to ratelimiting [ 18.682829] dracut-initqueue[583]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ 20.521357] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Reached target Remote File Systems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.333713] printk: systemd: 25 output lines suppressed due to ratelimiting [ 24.089743] SELinux: Disabled at runtime. [ 24.221513] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.235642] systemd[1]: Detected virtualization kvm. [ 24.240504] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.472305] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.477336] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.488516] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.494421] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.501848] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.517384] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.522977] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Apply Kernel Variables... [ 25.714252] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target Paths. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. Starting Flush Journal to Persistent Storage... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 27.004932] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 28.291627] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 28.369046] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 30.973028] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 31.119376] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 35.807927] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (10s / no limit)[ 36.219813] NFS: Registering the id_resolver key type [ 36.222396] Key type id_resolver registered [ 36.224639] Key type id_legacy registered [ **] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg302-server login: [ 63.899030] hrtimer: interrupt took 2039509 ns [ 100.048039] libcfs: loading out-of-tree module taints kernel. [ 100.076441] Key type ._llcrypt registered [ 100.078843] Key type .llcrypt registered [ 100.143586] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 116.124711] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 118.008349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 118.018164] alg: No test for adler32 (adler32-zlib) [ 119.217158] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 119.892161] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 121.576143] Key type lgssc registered [ 123.751382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 142.850677] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 184.679866] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 200.974613] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 201.017340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 202.339401] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 202.420446] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 202.486971] Lustre: lustre-MDT0000: new disk, initializing [ 202.542506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 202.554284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 207.740961] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.137027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 221.382686] Lustre: 6507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 221.420818] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 221.427833] Lustre: Skipped 1 previous similar message [ 221.508860] Lustre: lustre-MDT0001: new disk, initializing [ 221.562341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 221.593064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 221.606898] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 225.644785] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.561952] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 239.359222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 239.504418] Lustre: lustre-OST0000: new disk, initializing [ 239.507685] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 239.512675] Lustre: 8444:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 239.578906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 245.481848] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 246.832926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 246.843373] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 252.429095] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 259.914753] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 260.079495] Lustre: lustre-OST0001: new disk, initializing [ 260.082124] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 260.086422] Lustre: 9517:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 260.216390] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 265.497035] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 270.379169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 270.387729] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 270.446182] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 276.844639] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 288.302579] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 293.164346] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing check_logdir /tmp/testlogs/ [ 297.641315] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing yml_node [ 303.098847] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 305.390559] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 307.833856] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 309.480407] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 6 20:30:08 EDT 2026 [ 325.045605] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 327.352818] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 341.985371] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 341.988664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 342.005292] Lustre: Skipped 1 previous similar message [ 342.018312] Lustre: Skipped 1 previous similar message [ 347.106354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 347.110839] Lustre: Skipped 3 previous similar messages [ 352.224721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 352.239521] Lustre: Skipped 3 previous similar messages [ 356.320098] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 358.537340] Lustre: server umount lustre-MDT0000 complete [ 361.725772] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786062662 with bad export cookie 2002511757714614132 [ 361.727095] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 361.732683] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 362.468147] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 362.468389] LustreError: 6515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 362.470348] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 362.486255] Lustre: Skipped 3 previous similar messages [ 362.522055] LustreError: 6515:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 366.505172] LustreError: 6516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 366.505193] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 366.532961] LustreError: 6516:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 366.545266] Lustre: Skipped 2 previous similar messages [ 371.617333] LustreError: 6516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 371.629847] LustreError: 6516:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 376.288151] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 376.535528] Lustre: server umount lustre-MDT0001 complete [ 380.023894] Lustre: server umount lustre-OST0000 complete [ 383.517229] Lustre: server umount lustre-OST0001 complete [ 390.141195] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 398.335266] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 427.956986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 428.264456] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 428.286756] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 428.344956] Lustre: lustre-MDT0000: new disk, initializing [ 428.396116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 428.423218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 432.481941] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 441.713444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 441.784459] Lustre: 15075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 441.842572] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 441.847944] Lustre: Skipped 1 previous similar message [ 441.918989] Lustre: lustre-MDT0001: new disk, initializing [ 442.011523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 442.041983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 442.053035] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 446.189444] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 455.276506] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 460.020194] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 464.618607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 464.822635] Lustre: lustre-OST0000: new disk, initializing [ 464.826740] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 464.832255] Lustre: 17043:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 464.920146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 465.977748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 465.988736] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 466.142910] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 470.143020] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 478.250167] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 481.981671] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 482.184942] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 486.545436] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 486.799992] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 492.003418] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 492.005552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 492.008458] LustreError: Skipped 1 previous similar message [ 492.014213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 492.030585] Lustre: Skipped 1 previous similar message [ 492.032735] Lustre: Skipped 2 previous similar messages [ 494.787936] Lustre: server umount lustre-OST0000 complete [ 501.218587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 501.236364] Lustre: Skipped 1 previous similar message [ 505.453770] Lustre: server umount lustre-MDT0000 complete [ 506.337526] LustreError: 15958:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 506.347089] LustreError: 15958:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 508.815612] LustreError: 15066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786062809 with bad export cookie 2002511757714615665 [ 508.820525] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 508.824960] LustreError: 15066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 509.072329] Lustre: server umount lustre-MDT0001 complete [ 513.336805] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 20:33:32 (1786062812) [ 529.710459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 530.071507] LustreError: 19915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 530.140676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 533.470869] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 540.666040] LustreError: 19916:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 540.722955] LustreError: 19916:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 543.211771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 547.412608] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 554.650822] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 559.912590] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 561.996118] Lustre: Failing over lustre-MDT0000 [ 562.011142] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 562.229747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 562.243239] Lustre: Skipped 4 previous similar messages [ 562.382230] Lustre: server umount lustre-MDT0000 complete [ 567.265487] LustreError: 19916:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 567.287499] LustreError: 19916:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 581.794826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 581.911449] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 586.650713] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 587.270064] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 587.283044] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 587.294582] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 596.315783] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 602.593918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 602.627626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 602.660358] Lustre: Skipped 1 previous similar message [ 602.667192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 605.761943] Lustre: server umount lustre-MDT0000 complete [ 608.227862] LustreError: 19915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 608.256120] LustreError: 19915:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 609.329079] LustreError: 19897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786062910 with bad export cookie 2002511757714617079 [ 609.340856] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 609.346531] LustreError: 19897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 609.670739] Lustre: server umount lustre-MDT0001 complete [ 617.282252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 617.594224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 617.597210] Lustre: Skipped 2 previous similar messages [ 621.265755] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 629.768033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 635.321083] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 643.193581] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 648.790877] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 650.613595] Lustre: Failing over lustre-MDT0000 [ 650.722451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 650.730797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 650.740110] Lustre: Skipped 1 previous similar message [ 650.876351] Lustre: server umount lustre-MDT0000 complete [ 668.128588] Lustre: 3642:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786062953/real 1786062953] req@ffff98727e3b9c00 x1872822188300672/t0(0) o400->MGC192.168.203.102@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786062969 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 668.172418] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 669.627889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 673.249907] LustreError: 23249:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 673.262152] LustreError: 23249:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 678.377871] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x1bca59d5791f76a1 [ 678.386021] Lustre: MGC192.168.203.102@tcp: Connection restored to 0@lo (at 0@lo) [ 678.400391] Lustre: Skipped 1 previous similar message [ 678.796345] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 683.193718] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 684.040424] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 684.041037] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 684.174149] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 691.374952] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 698.849672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 698.853366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 698.884681] Lustre: Skipped 2 previous similar messages [ 698.977829] Lustre: Skipped 3 previous similar messages [ 701.714914] Lustre: server umount lustre-MDT0000 complete [ 705.797057] LustreError: 23231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786063006 with bad export cookie 2002511757714618017 [ 705.804618] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 705.806980] LustreError: 23231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 706.112298] Lustre: server umount lustre-MDT0001 complete [ 715.840334] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 20:36:55 (1786063015) [ 733.390587] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 741.890532] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 779.412955] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 790.271921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 790.477331] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 790.497980] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 790.556217] Lustre: lustre-MDT0000: new disk, initializing [ 790.611978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 790.617324] Lustre: Skipped 2 previous similar messages [ 790.631550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 794.850206] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 805.005169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 805.122576] Lustre: 29625:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 805.323293] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 810.401677] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 819.376731] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 826.978731] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 835.863496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 836.071708] Lustre: 31593:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 837.784189] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 837.863258] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 837.889872] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 837.919467] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 842.597212] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 852.541611] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 858.407492] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 879.082178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 879.096871] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 879.105545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 879.105553] Lustre: Skipped 2 previous similar messages [ 879.110376] Lustre: Skipped 1 previous similar message [ 881.810256] Lustre: server umount lustre-OST0000 complete [ 887.786372] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 892.738366] Lustre: server umount lustre-MDT0000 complete [ 894.980468] LustreError: 31601:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 894.991199] LustreError: 31601:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 896.785631] LustreError: 30420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786063197 with bad export cookie 2002511757714618423 [ 896.795217] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 896.798360] LustreError: 30420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 897.171893] Lustre: server umount lustre-MDT0001 complete [ 907.651357] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 909.821966] Key type lgssc unregistered [ 910.029723] LNet: 33576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.035825] LNetError: 33576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 910.046987] LNet: Removed LNI 192.168.203.102@tcp [ 910.840149] Key type .llcrypt unregistered [ 910.844553] Key type ._llcrypt unregistered [ 936.367864] Key type ._llcrypt registered [ 936.370355] Key type .llcrypt registered [ 936.463195] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 937.683643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 937.884429] alg: No test for adler32 (adler32-zlib) [ 939.397566] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 939.614616] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 941.400232] Key type lgssc registered [ 942.738269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 951.473085] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 951.498687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 952.851486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 957.719449] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 962.090491] LustreError: 34993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 962.106296] LustreError: 34993:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 966.235207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 966.522752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 972.143235] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 980.112226] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 987.071485] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 993.806389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 994.038921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 998.169353] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1001.100379] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1007.506977] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1011.464769] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1025.441487] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 20:42:04 (1786063324) [ 1028.144803] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1029.941814] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1039.077549] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 20:42:17 (1786063337) [ 1047.378830] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 20:42:26 (1786063346) [ 1048.530986] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1057.187962] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1057.197198] Lustre: Skipped 2 previous similar messages [ 1058.901813] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 20:42:38 (1786063358) [ 1063.590994] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1063.594868] Lustre: Skipped 3 previous similar messages [ 1071.728605] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1071.735130] Lustre: Skipped 3 previous similar messages [ 1086.163580] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 20:43:05 (1786063385) [ 1088.887803] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1092.018058] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1100.824872] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 20:43:19 (1786063399) [ 1103.030529] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1103.034326] Lustre: Skipped 6 previous similar messages [ 1116.933763] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 20:43:36 (1786063416) [ 1131.492207] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1131.501932] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1131.510422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1134.579121] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1134.584466] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1134.592302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1137.461403] Lustre: server umount lustre-OST0000 complete [ 1144.801686] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1144.803561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1144.821878] Lustre: Skipped 1 previous similar message [ 1144.826764] Lustre: Skipped 1 previous similar message [ 1147.581023] Lustre: server umount lustre-MDT0000 complete [ 1149.922392] LustreError: 34993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1149.934475] LustreError: 34993:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1151.127442] LustreError: 39058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786063452 with bad export cookie 3809529310452148956 [ 1151.140653] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1151.144229] LustreError: 39058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1151.411716] Lustre: server umount lustre-MDT0001 complete [ 1162.128913] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 1164.419909] Key type lgssc unregistered [ 1164.993907] LNet: 42855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1165.001369] LNetError: 42855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1165.013963] LNet: Removed LNI 192.168.203.102@tcp [ 1165.767567] Key type .llcrypt unregistered [ 1165.770785] Key type ._llcrypt unregistered [ 1184.331965] Key type ._llcrypt registered [ 1184.334402] Key type .llcrypt registered [ 1184.419511] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 1197.184398] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 1199.535417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1199.562680] alg: No test for adler32 (adler32-zlib) [ 1200.649801] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 1200.944428] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 1202.640167] Key type lgssc registered [ 1205.001815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1241.701653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1241.731206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1242.948357] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1242.968431] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1243.026731] Lustre: lustre-MDT0000: new disk, initializing [ 1243.080390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1243.095395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1247.593731] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1257.221853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1257.325707] Lustre: 45988:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1257.367139] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1257.372580] Lustre: Skipped 1 previous similar message [ 1257.452403] Lustre: lustre-MDT0001: new disk, initializing [ 1257.563266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1257.609362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1257.644731] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1262.509678] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1273.286497] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1279.828468] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1286.886435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1287.154599] Lustre: lustre-OST0000: new disk, initializing [ 1287.163190] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1287.183989] Lustre: 47958:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1287.336488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1293.138635] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1296.443903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1296.449324] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1296.548246] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1303.349432] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1308.342827] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1308.540753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1312.540652] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1312.848658] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1316.835611] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1316.842866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1316.855745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1317.354964] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1317.372084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1317.373682] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1317.435653] Lustre: Skipped 1 previous similar message [ 1322.523718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1322.530104] Lustre: Skipped 1 previous similar message [ 1327.606628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1327.615356] Lustre: Skipped 1 previous similar message [ 1329.120131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1329.209641] Lustre: server umount lustre-OST0000 complete [ 1334.242523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1334.248940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1334.259923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1337.854774] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1339.646275] Lustre: server umount lustre-MDT0000 complete [ 1343.704417] LustreError: 45981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786063644 with bad export cookie 12570990668324175789 [ 1343.707788] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1343.711792] LustreError: 45981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1344.041957] Lustre: server umount lustre-MDT0001 complete [ 1354.268431] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 20:47:33 (1786063653) [ 1362.303192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1362.799088] LustreError: 50421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1362.825050] LustreError: 50421:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1362.906940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1366.980641] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1368.036571] LustreError: 50422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1372.131461] LustreError: 50421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1374.161566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1374.518334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1378.717647] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1386.828771] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1393.044968] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1398.981134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1399.196539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1404.703149] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1412.587366] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1417.356254] LustreError: 52949:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1418.482265] Lustre: Modifying parameter general.timeout=1 in log params [ 1419.548512] Lustre: Modifying parameter general.timeout=2 in log params [ 1421.600658] Lustre: Modifying parameter general.timeout=5 in log params [ 1421.604190] Lustre: Skipped 2 previous similar messages [ 1425.616773] Lustre: Modifying parameter general.timeout=10 in log params [ 1425.621519] Lustre: Skipped 4 previous similar messages [ 1434.106664] Lustre: Modifying parameter general.timeout=19 in log params [ 1434.109727] Lustre: Skipped 8 previous similar messages [ 1450.870488] Lustre: Modifying parameter general.timeout=37 in log params [ 1450.882737] Lustre: Skipped 17 previous similar messages [ 1483.003772] Lustre: Modifying parameter general.timeout=70 in log params [ 1483.011600] Lustre: Skipped 32 previous similar messages [ 1546.448557] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 20:50:45 (1786063845) [ 1547.719428] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1547.738797] Lustre: Skipped 60 previous similar messages [ 1550.077467] Lustre: 59892:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1551.764828] Lustre: 59990:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1553.977324] Lustre: 60089:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1559.738487] Lustre: 60403:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1559.746415] Lustre: 60403:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1584.869093] Lustre: 61767:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1584.909063] LustreError: 61767:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1588.423981] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 20:51:27 (1786063887) [ 1599.971728] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1599.975814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1599.983717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1599.988872] Lustre: Skipped 2 previous similar messages [ 1613.312120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1613.406721] Lustre: server umount lustre-MDT0000 complete [ 1613.796870] LustreError: 50421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1613.812339] LustreError: 50421:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1616.578921] LustreError: 50401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786063917 with bad export cookie 12570990668324176671 [ 1616.585226] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1616.592239] LustreError: 50401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1616.790272] Lustre: server umount lustre-MDT0001 complete [ 1630.399195] Lustre: server umount lustre-OST0000 complete [ 1653.813550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1654.056444] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1654.351160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1658.462868] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1667.082390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1667.154302] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1667.174692] Lustre: 64300:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1671.707503] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1678.678468] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1684.444459] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1690.242429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1690.401145] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1690.608736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1690.613839] Lustre: Skipped 1 previous similar message [ 1696.164942] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1703.297706] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1704.999397] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1705.009797] Lustre: Skipped 22 previous similar messages [ 1705.066636] Lustre: 66867:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1714.150213] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1714.157701] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1714.186673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1714.188670] Lustre: Skipped 3 previous similar messages [ 1714.219086] Lustre: Skipped 7 previous similar messages [ 1717.056972] Lustre: server umount lustre-OST0000 complete [ 1723.878993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1723.898262] LustreError: Skipped 1 previous similar message [ 1727.831569] Lustre: server umount lustre-MDT0000 complete [ 1729.515232] LustreError: 64309:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1729.529216] LustreError: 64309:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1731.493187] LustreError: 64293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786064032 with bad export cookie 12570990668324179737 [ 1731.499047] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1731.506223] LustreError: 64293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1731.848490] Lustre: server umount lustre-MDT0001 complete [ 1741.864540] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 1744.230937] Key type lgssc unregistered [ 1744.458798] LNet: 68082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1744.468250] LNetError: 68082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1744.483214] LNet: Removed LNI 192.168.203.102@tcp [ 1745.039995] Key type .llcrypt unregistered [ 1745.049705] Key type ._llcrypt unregistered [ 1769.676445] Key type ._llcrypt registered [ 1769.678542] Key type .llcrypt registered [ 1769.789245] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 1771.149752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1771.171073] alg: No test for adler32 (adler32-zlib) [ 1772.339620] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 1772.502451] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 1774.128158] Key type lgssc registered [ 1775.211214] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1783.846742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1783.858262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1785.168492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1789.216726] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1795.556953] LustreError: 69493:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1795.568217] LustreError: 69493:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1797.434969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1798.073645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1802.117743] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1807.737854] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1813.035478] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1818.365092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1818.768393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1824.061142] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1830.408923] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1838.152596] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1845.734294] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1845.737774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1845.756589] Lustre: Skipped 2 previous similar messages [ 1845.768216] Lustre: Skipped 2 previous similar messages [ 1850.851106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1850.857281] Lustre: Skipped 2 previous similar messages [ 1851.414384] Lustre: server umount lustre-MDT0000 complete [ 1855.000299] LustreError: 69476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786064156 with bad export cookie 124153804605774242 [ 1855.010662] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1855.014736] LustreError: 69476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1855.237781] Lustre: server umount lustre-MDT0001 complete [ 1868.556524] Lustre: server umount lustre-OST0000 complete [ 1894.035286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1894.228950] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1894.260390] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1894.518096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1898.783796] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1906.242390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1906.330844] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1906.350494] Lustre: 74366:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1906.698605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1910.278880] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1917.096637] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1923.018407] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1932.514970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1932.674883] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1933.185224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1939.810367] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 1947.795510] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1949.299880] Lustre: 76938:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1950.870850] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1958.881736] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1958.886209] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1958.894415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1959.398066] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1961.262848] Lustre: server umount lustre-OST0000 complete [ 1968.099911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1968.104315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1968.111776] Lustre: Skipped 1 previous similar message [ 1968.115169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1968.118755] Lustre: Skipped 2 previous similar messages [ 1974.843542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1974.850048] Lustre: Skipped 3 previous similar messages [ 1979.360196] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1979.461485] Lustre: server umount lustre-MDT0000 complete [ 1979.883417] LustreError: 74377:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1979.894867] LustreError: 74377:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1983.027104] LustreError: 76940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786064284 with bad export cookie 124153804605775096 [ 1983.027736] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1983.037647] LustreError: 76940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1983.300580] Lustre: server umount lustre-MDT0001 complete [ 1995.932285] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 1997.938845] Key type lgssc unregistered [ 1998.170466] LNet: 78153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1998.175421] LNetError: 78153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1998.191134] LNet: Removed LNI 192.168.203.102@tcp [ 1999.133612] Key type .llcrypt unregistered [ 1999.141886] Key type ._llcrypt unregistered [ 2026.142300] Key type ._llcrypt registered [ 2026.144892] Key type .llcrypt registered [ 2026.282539] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 2027.542057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2027.560516] alg: No test for adler32 (adler32-zlib) [ 2028.516572] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 2028.769244] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 2030.568178] Key type lgssc registered [ 2031.649731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2038.909517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2038.924803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2040.334744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2043.916447] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2050.534516] LustreError: 79570:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2050.546713] LustreError: 79570:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2051.131062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2051.622220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2055.691784] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2061.991365] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2067.413934] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2073.116843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2073.744471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2079.395480] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2087.070763] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2092.450365] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2093.368139] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2095.232575] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2095.237164] Lustre: Skipped 1 previous similar message [ 2097.762848] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2097.766646] Lustre: Skipped 2 previous similar messages [ 2102.502187] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2102.508265] Lustre: Skipped 4 previous similar messages [ 2110.867291] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2110.875249] Lustre: Skipped 8 previous similar messages [ 2126.888832] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2126.892702] Lustre: Skipped 18 previous similar messages [ 2158.902767] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2158.906600] Lustre: Skipped 34 previous similar messages [ 2191.303620] Lustre: 87685:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2194.184143] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 21:01:33 (1786064493) [ 2195.598219] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2197.510552] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 21:01:36 (1786064496) [ 2203.109987] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2203.111514] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2203.124338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2203.134054] Lustre: Skipped 1 previous similar message [ 2206.847567] Lustre: server umount lustre-OST0000 complete [ 2210.319702] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2210.326249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2210.334910] Lustre: Skipped 1 previous similar message [ 2210.341264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2213.349669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2213.360727] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2218.465602] LustreError: 79565:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2218.478706] LustreError: 79565:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2218.510148] Lustre: server umount lustre-MDT0000 complete [ 2221.509156] LustreError: 79550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786064522 with bad export cookie 2317522536747986296 [ 2221.522759] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2221.527776] LustreError: 79550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2221.816340] Lustre: server umount lustre-MDT0001 complete [ 2232.442202] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 2235.096409] Key type lgssc unregistered [ 2235.323643] LNet: 88939:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2235.336530] LNetError: 88939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2235.359645] LNet: Removed LNI 192.168.203.102@tcp [ 2236.011535] Key type .llcrypt unregistered [ 2236.017405] Key type ._llcrypt unregistered [ 2240.877938] Key type ._llcrypt registered [ 2240.882868] Key type .llcrypt registered [ 2241.025703] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2246.117400] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules [ 2247.266962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2247.295117] alg: No test for adler32 (adler32-zlib) [ 2248.524396] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 2248.946199] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 2250.602094] Key type lgssc registered [ 2251.568605] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2253.446769] LustreError: 90140:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2270.605283] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 2276.369143] LustreError: 90140:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2280.329265] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2280.368886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2281.708155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2285.431808] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2287.073640] LustreError: 91165:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2287.109870] LustreError: 91165:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2291.172346] LustreError: 91166:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2294.211241] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 21:03:13 (1786064593) [ 2296.293793] LustreError: 91165:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2299.746201] LustreError: 91197:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2299.755283] LustreError: 91197:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2299.760538] LustreError: 91197:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 2300.015430] Lustre: server umount lustre-MDT0000 complete [ 2313.387675] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 2316.661682] Key type lgssc unregistered [ 2317.128803] LNet: 92694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2317.134702] LNetError: 92694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2317.152846] LNet: Removed LNI 192.168.203.102@tcp [ 2317.914680] Key type .llcrypt unregistered [ 2317.916731] Key type ._llcrypt unregistered [ 2335.660490] Key type ._llcrypt registered [ 2335.664709] Key type .llcrypt registered [ 2335.807771] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 2336.933416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2337.041115] alg: No test for adler32 (adler32-zlib) [ 2338.357973] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 2338.599832] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 2340.256171] Key type lgssc registered [ 2341.250946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2347.736904] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2347.749863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2349.034573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2352.650213] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2354.147718] LustreError: 93912:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2354.170315] LustreError: 93912:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2359.278065] LustreError: 93913:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2360.294267] LustreError: 93912:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2363.613144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2364.532580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2370.305960] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2376.507811] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2382.840985] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2389.790930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2390.454563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2398.810315] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2406.401217] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2455.733940] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 21:05:54 (1786064754) [ 2457.931460] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2459.750978] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 21:05:58 (1786064758) [ 2466.278256] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2466.286985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2466.309362] Lustre: Skipped 1 previous similar message [ 2469.015530] Lustre: server umount lustre-MDT0000 complete [ 2471.417777] LustreError: 93909:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2471.448720] LustreError: 93909:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2473.204530] LustreError: 93894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786064774 with bad export cookie 11886246513325837269 [ 2473.214141] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2473.418311] Lustre: server umount lustre-MDT0001 complete [ 2486.361594] Lustre: server umount lustre-OST0000 complete [ 2496.327641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2496.682435] LustreError: 98190:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2496.837703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2500.683483] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2506.215039] LustreError: 98191:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2506.235963] LustreError: 98191:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2508.367637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2508.993114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2513.321833] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2519.665402] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2525.024918] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2537.399092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2537.527399] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2537.535378] LustreError: 98177:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2537.553392] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2537.560706] LustreError: 100274:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2537.565436] LustreError: 100274:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2537.569675] LustreError: 100274:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2537.613754] Lustre: server umount lustre-OST0000 complete [ 2537.616794] LustreError: 100274:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2546.047100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2546.174763] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2546.180840] LustreError: 98180:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2546.199901] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2546.206765] LustreError: 100741:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2546.211321] LustreError: 100741:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2546.216579] LustreError: 100741:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2546.263409] Lustre: server umount lustre-OST0000 complete [ 2546.265728] LustreError: 100741:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2556.145749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2556.281829] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2556.288483] Lustre: Found index 0 for lustre-OST0000, updating log [ 2556.294864] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2556.352482] Lustre: lustre-OST0000: new disk, initializing [ 2556.356792] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2556.363183] Lustre: 101275:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2556.562996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2562.729967] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2566.278957] Lustre: Failing over lustre-OST0000 [ 2566.414694] Lustre: server umount lustre-OST0000 complete [ 2572.769963] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2572.773951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2572.779137] Lustre: Skipped 2 previous similar messages [ 2572.791393] Lustre: Skipped 2 previous similar messages [ 2576.614484] Lustre: server umount lustre-MDT0000 complete [ 2577.891062] LustreError: 98186:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2577.959838] LustreError: 98186:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2581.007973] LustreError: 98172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786064882 with bad export cookie 11886246513325840013 [ 2581.024849] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2581.028937] LustreError: 98172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2581.342918] Lustre: server umount lustre-MDT0001 complete [ 2589.461606] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 21:08:08 (1786064888) [ 2604.524866] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 2614.103396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2614.394104] LustreError: 103619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2614.417452] LustreError: 103619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2614.508550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2618.940616] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2626.158816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2632.016199] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2635.138796] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2641.685978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2647.914557] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2656.750753] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2656.901772] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2657.136909] Lustre: lustre-OST0001: new disk, initializing [ 2657.140426] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2657.145511] Lustre: 105938:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2657.357708] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2657.369509] Lustre: Skipped 2 previous similar messages [ 2663.005551] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2663.120459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2663.150895] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2663.471402] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2670.421949] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2678.025764] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2683.370423] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2683.380871] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2683.410199] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2683.416967] Lustre: Skipped 1 previous similar message [ 2686.250207] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.2@tcp (stopping) [ 2686.261386] Lustre: Skipped 2 previous similar messages [ 2688.478466] Lustre: server umount lustre-MDT0001 complete [ 2689.001632] LustreError: 103620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2689.056347] LustreError: 103620:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2697.908438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2698.058669] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2698.066636] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2698.146910] Lustre: 103607:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2700.148997] LustreError: 107485:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2700.164278] LustreError: 107485:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2700.184940] LustreError: 103611:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2700.294542] LustreError: 107490:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.102@tcp: cfg command failed: rc = -17 [ 2700.305954] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2700.305954] [ 2703.870595] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2704.000303] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2712.942323] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 21:10:12 (1786065012) [ 2729.253557] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 2747.579457] Lustre: 109550:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2773.000508] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2776.343086] Lustre: 110685:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2862.563611] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2862.563961] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2862.573074] Lustre: Skipped 2 previous similar messages [ 2862.575327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2862.582813] Lustre: Skipped 3 previous similar messages [ 2866.166889] Lustre: server umount lustre-MDT0000 complete [ 2867.680917] LustreError: 103620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2867.688514] LustreError: 103620:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 2869.503546] LustreError: 103601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786065170 with bad export cookie 11886246513325841007 [ 2869.505259] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2869.510129] LustreError: 103601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2869.697746] Lustre: server umount lustre-MDT0001 complete [ 2884.876444] Lustre: server umount lustre-OST0000 complete [ 2888.801562] Lustre: 93225:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786065173/real 1786065173] req@ffff9871546f4a80 x1872824515259008/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786065189 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2888.885110] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2888.970297] Lustre: Skipped 3 previous similar messages [ 2890.261824] Lustre: server umount lustre-OST0001 complete [ 2895.603506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2904.236470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2913.321942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2922.557347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2942.136578] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 2952.463268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2952.478686] Lustre: lustre-MDT0000: reset Object Index mappings [ 2952.854208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2952.861490] Lustre: Skipped 1 previous similar message [ 2956.778416] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2964.698698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2964.715612] Lustre: lustre-MDT0001: reset Object Index mappings [ 2969.547344] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2972.104206] Lustre: 115782:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2978.493316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2984.718474] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 2990.445726] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 2992.350754] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2997.048882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 2997.063604] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2997.238895] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2999.028970] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 3006.783643] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3010.683761] Lustre: 117663:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3055.355982] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 21:15:54 (1786065354) [ 3082.724234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3082.728083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3082.737609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3087.790724] Lustre: server umount lustre-MDT0000 complete [ 3090.634881] LustreError: 114619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786065391 with bad export cookie 11886246513325887858 [ 3090.646027] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3090.661246] LustreError: 114619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3090.825637] LustreError: 118214:0:(ldlm_resource.c:1207:ldlm_resource_complain()) mdt-lustre-MDT0001_UUID: namespace resource [0x240004a50:0x19:0x0].0x0 (ffff98727ccb4200) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3100.642288] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3100.652141] Lustre: Skipped 9 previous similar messages [ 3105.248112] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3105.662613] Lustre: server umount lustre-MDT0001 complete [ 3123.168481] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3123.248780] Lustre: server umount lustre-OST0000 complete [ 3132.752540] Lustre: server umount lustre-OST0001 complete [ 3138.951402] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 3149.402233] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 3184.239708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3184.406205] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3184.431440] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3184.489240] Lustre: lustre-MDT0000: new disk, initializing [ 3184.547947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3184.553091] Lustre: Skipped 3 previous similar messages [ 3184.565328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3189.150537] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 3195.984864] Lustre: server umount lustre-MDT0000 complete [ 3205.094369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3205.353264] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3205.362385] Lustre: 121927:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3205.373777] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3209.338300] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 3214.094914] Lustre: server umount lustre-MDT0000 complete [ 3221.950873] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 21:18:40 (1786065520) [ 3224.429264] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3226.774963] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 21:18:45 (1786065525) [ 3242.359671] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 3252.430703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3257.538165] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 3264.501588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3264.594736] Lustre: 123978:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3264.653532] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3264.657574] Lustre: Skipped 1 previous similar message [ 3264.718163] Lustre: lustre-MDT0001: new disk, initializing [ 3264.837555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3264.865750] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3268.781210] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 3272.981637] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3279.892475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3280.242183] Lustre: lustre-OST0000: new disk, initializing [ 3280.245124] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3280.250299] Lustre: 125553:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3282.079215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3282.096623] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3282.288137] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3287.303090] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 3296.976102] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3297.161697] Lustre: lustre-OST0001: new disk, initializing [ 3297.165610] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3297.171737] Lustre: 126425:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3298.522656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3298.543737] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3298.662497] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3303.709253] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 3312.129466] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3315.204905] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3697.526759] Lustre: 132833:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3697.533701] Lustre: 132833:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3697.538773] Lustre: 132833:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3697.544588] Lustre: 132833:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3697.550617] Lustre: 132833:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3697.557287] Lustre: 132833:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3986.294075] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 21:31:25 (1786066285) [ 4001.763685] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4001.770472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4001.776855] Lustre: Skipped 7 previous similar messages [ 4001.790044] Lustre: Skipped 2 previous similar messages [ 4006.495390] Lustre: server umount lustre-MDT0000 complete [ 4006.895868] LustreError: 123984:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4006.921930] LustreError: 123984:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 4009.879247] LustreError: 132871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786066310 with bad export cookie 11886246513325953973 [ 4009.880423] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4009.906921] LustreError: 132871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4012.011470] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4012.015588] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4012.061508] Lustre: Skipped 2 previous similar messages [ 4012.083323] Lustre: Skipped 1 previous similar message [ 4024.288386] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4024.290055] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4024.311389] Lustre: Skipped 5 previous similar messages [ 4024.613705] Lustre: server umount lustre-MDT0001 complete [ 4034.645169] Lustre: server umount lustre-OST0000 complete [ 4043.932212] Lustre: server umount lustre-OST0001 complete [ 4050.257970] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 4058.441719] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 4093.582826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4093.781146] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4093.802712] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4093.862707] Lustre: lustre-MDT0000: new disk, initializing [ 4093.944438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4093.951209] Lustre: Skipped 5 previous similar messages [ 4093.970204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4097.971668] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 4109.498543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4109.850709] Lustre: 135886:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4109.900644] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4109.904230] Lustre: Skipped 1 previous similar message [ 4109.963569] Lustre: lustre-MDT0001: new disk, initializing [ 4110.221562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4110.312231] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4114.291081] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 4122.992709] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4128.141447] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4136.585639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4136.744314] Lustre: lustre-OST0000: new disk, initializing [ 4136.747304] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4136.751357] Lustre: 137855:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4136.816989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4136.832062] Lustre: Skipped 1 previous similar message [ 4138.354722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4138.370306] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4138.768349] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4143.134086] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 4152.436169] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4160.704234] Lustre: lustre-MDD0000: changelog on [ 4163.411856] Lustre: lustre-MDD0001: changelog on [ 5925.496729] Lustre: lustre-MDD0001: changelog off [ 5927.543217] Lustre: 143613:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5928.413215] Lustre: lustre-MDD0000: changelog off [ 5936.099785] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5936.100153] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5936.107445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5936.116090] Lustre: Skipped 1 previous similar message [ 5940.950660] Lustre: server umount lustre-OST0000 complete [ 5946.337501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5946.358775] Lustre: Skipped 1 previous similar message [ 5946.365760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5946.369454] Lustre: Skipped 1 previous similar message [ 5951.457886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5951.461413] Lustre: Skipped 1 previous similar message [ 5951.700580] Lustre: server umount lustre-MDT0000 complete [ 5955.568129] LustreError: 135878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786068256 with bad export cookie 11886246513326771174 [ 5955.573444] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5955.581818] LustreError: 135878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5955.831063] Lustre: server umount lustre-MDT0001 complete [ 5967.998113] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 5970.060742] Key type lgssc unregistered [ 5970.386791] LNet: 146998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5970.392362] LNetError: 146998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5970.409630] LNet: Removed LNI 192.168.203.102@tcp [ 5971.304219] Key type .llcrypt unregistered [ 5971.306885] Key type ._llcrypt unregistered [ 5992.141278] Key type ._llcrypt registered [ 5992.146367] Key type .llcrypt registered [ 5992.245350] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 6008.855099] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 6011.873060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6011.888397] alg: No test for adler32 (adler32-zlib) [ 6012.906971] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 6013.525955] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 6015.392163] Key type lgssc registered [ 6016.206766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6056.632380] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6056.657442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6057.982319] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6058.002781] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6058.064655] Lustre: lustre-MDT0000: new disk, initializing [ 6058.162535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6058.202827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6063.525436] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6074.508449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6074.618756] Lustre: 150200:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6074.662456] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6074.665966] Lustre: Skipped 1 previous similar message [ 6074.916757] Lustre: lustre-MDT0001: new disk, initializing [ 6075.051048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6075.149258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6075.190937] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6079.293906] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6090.005496] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6096.688094] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6104.688695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6104.978508] Lustre: lustre-OST0000: new disk, initializing [ 6104.981698] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6104.988389] Lustre: 152170:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6105.170944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6111.152333] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6114.928517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6114.966228] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6115.278337] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6122.000480] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6128.371692] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_module obdecho/obdecho [ 6130.475564] LustreError: 153177:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6130.484247] LustreError: 153177:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6132.689990] LustreError: 153274:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6146.053384] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6146.072578] Lustre: Skipped 1 previous similar message [ 6146.077544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6146.082285] Lustre: Skipped 1 previous similar message [ 6151.140033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6151.227030] Lustre: Skipped 2 previous similar messages [ 6152.367507] Lustre: server umount lustre-MDT0000 complete [ 6157.324061] LustreError: 150195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786068458 with bad export cookie 16835529319200136121 [ 6157.325798] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6157.330028] LustreError: 150195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6157.822548] Lustre: server umount lustre-MDT0001 complete [ 6172.994760] Lustre: server umount lustre-OST0000 complete [ 6193.714428] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 6204.711796] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 6245.629203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6245.839480] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6245.868659] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6245.921580] Lustre: lustre-MDT0000: new disk, initializing [ 6245.970305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6245.982483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6250.198123] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6259.957085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6260.051335] Lustre: 156989:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6260.094544] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6260.098589] Lustre: Skipped 1 previous similar message [ 6260.172806] Lustre: lustre-MDT0001: new disk, initializing [ 6260.267841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6260.301386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6260.345952] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6265.186329] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6274.744972] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6280.334787] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6287.754855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6288.281223] Lustre: lustre-OST0000: new disk, initializing [ 6288.285922] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6288.296245] Lustre: 158962:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6288.437876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6294.516948] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6298.194483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6298.209951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6298.282669] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6305.102842] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6311.465769] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6311.904324] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6317.824380] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6318.234677] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6324.195873] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6324.197161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6324.198703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6324.208767] Lustre: Skipped 2 previous similar messages [ 6324.226446] LustreError: Skipped 1 previous similar message [ 6327.118939] Lustre: server umount lustre-OST0000 complete [ 6331.874731] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6331.879645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6331.889366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6331.893864] Lustre: Skipped 1 previous similar message [ 6334.469047] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6339.552744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6339.561288] Lustre: Skipped 3 previous similar messages [ 6346.208128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6346.552757] Lustre: server umount lustre-MDT0000 complete [ 6349.795082] LustreError: 157002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6349.811994] LustreError: 157002:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6351.227701] LustreError: 156982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786068652 with bad export cookie 16835529319200137080 [ 6351.234148] LustreError: 156982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6351.234580] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6351.597103] Lustre: server umount lustre-MDT0001 complete [ 6356.687853] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 22:10:54 (1786068654) [ 6358.701347] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6359.958614] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 22:10:59 (1786068659) [ 6368.123898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6368.418477] LustreError: 161376:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6368.474777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6373.036213] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6373.858334] LustreError: 161377:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6378.979265] LustreError: 161376:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6382.156975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6387.149243] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6395.035817] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6400.387559] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6406.825671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6407.032099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6407.036193] Lustre: Skipped 1 previous similar message [ 6412.570465] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6420.551220] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6428.649311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6428.657511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6428.669929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6428.678074] Lustre: Skipped 2 previous similar messages [ 6429.202530] Lustre: server umount lustre-MDT0000 complete [ 6429.676529] LustreError: 161377:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6429.718561] LustreError: 161377:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6433.225886] LustreError: 162111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786068734 with bad export cookie 16835529319200137962 [ 6433.230474] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6433.235562] LustreError: 162111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6438.818204] LustreError: 161372:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6438.842349] LustreError: 161372:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6439.494585] Lustre: server umount lustre-MDT0001 complete [ 6445.507550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6457.989280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6458.196778] LustreError: 165216:0:(llog.c:1655:llog_backup()) MGC192.168.203.102@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6458.206346] Lustre: 165216:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.102@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6462.105210] LustreError: 165221:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6494.177422] LustreError: 165222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6494.204250] LustreError: 165222:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 26 previous similar messages [ 6519.265569] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6519.275469] Lustre: 165216:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.102@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6558.422939] LustreError: 165241:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6558.433082] LustreError: 165241:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 39 previous similar messages [ 6560.270027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6564.634418] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6565.348133] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6572.930816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6579.680770] LustreError: 148423:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b336ad52 x1872828368553344/t0(0) o250->MGC192.168.203.102@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6583.447280] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6585.367612] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6590.309800] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6592.219881] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6598.038733] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6599.155542] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6621.157480] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6621.166646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6621.167473] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6621.182657] Lustre: Skipped 5 previous similar messages [ 6621.189746] Lustre: Skipped 4 previous similar messages [ 6631.392127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6631.612367] Lustre: server umount lustre-MDT0000 complete [ 6636.026354] LustreError: 161357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786068937 with bad export cookie 16835529319200144416 [ 6636.031042] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6636.034349] LustreError: 161357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6636.238745] Lustre: server umount lustre-MDT0001 complete [ 6650.453840] Lustre: server umount lustre-OST0000 complete [ 6671.333507] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 6680.934872] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 6715.037553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6715.242730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6715.282631] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6715.332958] Lustre: lustre-MDT0000: new disk, initializing [ 6715.376503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6715.383443] Lustre: Skipped 1 previous similar message [ 6715.392876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6719.326594] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6729.654313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6729.911589] Lustre: 170320:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6730.195826] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6734.246900] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6744.137770] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6750.735637] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6757.972071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6758.343809] Lustre: lustre-OST0000: new disk, initializing [ 6758.351256] Lustre: Skipped 1 previous similar message [ 6758.360395] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6758.371878] Lustre: Skipped 2 previous similar messages [ 6758.387872] Lustre: 172289:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6765.291299] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6766.636983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6766.646519] Lustre: Skipped 1 previous similar message [ 6766.656452] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6766.792494] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6774.929465] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6780.229480] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6780.437773] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6785.625662] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6785.804411] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6792.677895] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6792.681438] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6792.683833] LustreError: Skipped 1 previous similar message [ 6792.685233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6792.685240] Lustre: Skipped 7 previous similar messages [ 6792.703806] Lustre: Skipped 1 previous similar message [ 6794.284751] Lustre: server umount lustre-OST0000 complete [ 6801.889472] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6804.872894] Lustre: server umount lustre-MDT0000 complete [ 6808.046223] LustreError: 170330:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6808.057703] LustreError: 170330:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 22 previous similar messages [ 6809.377511] LustreError: 170313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786069110 with bad export cookie 16835529319200161251 [ 6809.382580] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6809.402689] LustreError: 170313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6809.748390] Lustre: server umount lustre-MDT0001 complete [ 6818.959993] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 22:18:38 (1786069118) [ 6825.058818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6830.309151] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6839.857398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6844.254767] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6850.847471] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6855.608891] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6861.187659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6868.452457] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6875.638648] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6886.038750] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 22:19:45 (1786069185) [ 6891.494049] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6891.497037] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6891.499280] LustreError: Skipped 1 previous similar message [ 6891.508264] Lustre: Skipped 2 previous similar messages [ 6891.517632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6891.530983] Lustre: Skipped 5 previous similar messages [ 6895.595294] Lustre: server umount lustre-OST0000 complete [ 6905.750533] Lustre: server umount lustre-MDT0000 complete [ 6909.125109] LustreError: 174685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786069210 with bad export cookie 16835529319200162133 [ 6909.125919] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6909.136165] LustreError: 174685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6909.434166] Lustre: server umount lustre-MDT0001 complete [ 6920.154936] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 6922.176661] Key type lgssc unregistered [ 6922.363989] LNet: 178434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6922.371641] LNetError: 178434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6922.395978] LNet: Removed LNI 192.168.203.102@tcp [ 6923.045951] Key type .llcrypt unregistered [ 6923.048752] Key type ._llcrypt unregistered [ 6940.857782] Key type ._llcrypt registered [ 6940.859860] Key type .llcrypt registered [ 6940.954630] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 6941.926246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6942.045231] alg: No test for adler32 (adler32-zlib) [ 6943.388543] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 6943.554965] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 6945.232200] Key type lgssc registered [ 6946.381963] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6953.735522] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6953.751539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6969.248378] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6974.432733] LustreError: 179679:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.102@tcp: failed processing log, type 4: rc = -110 [ 6979.552501] LustreError: 179679:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6979.558939] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6979.561995] LustreError: MGC192.168.203.102@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6979.568744] LustreError: 179679:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6979.573500] LustreError: 179679:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 6979.578444] LustreError: 179679:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 6979.611117] Lustre: server umount lustre-OST0000 complete [ 6979.613670] LustreError: 179679:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6987.923254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6989.384711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6994.410395] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 6999.547845] LustreError: 180178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6999.568514] LustreError: 180178:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7002.522059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7002.908557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7007.098961] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7014.962475] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7020.990149] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7027.182066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7027.298935] LustreError: 181993:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7027.306034] Lustre: 181993:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.102@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7027.515326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7033.960804] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7041.241752] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7044.070151] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7044.100767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7044.108259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7045.091839] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7045.092424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7045.101620] Lustre: Skipped 1 previous similar message [ 7045.104978] Lustre: Skipped 1 previous similar message [ 7050.217079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7050.220243] Lustre: Skipped 1 previous similar message [ 7055.332068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7055.335870] Lustre: Skipped 2 previous similar messages [ 7058.400142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7058.516271] Lustre: server umount lustre-MDT0000 complete [ 7060.456516] LustreError: 180175:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7060.473967] LustreError: 180175:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7061.914547] LustreError: 180160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786069362 with bad export cookie 16308770956145131207 [ 7061.922560] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7061.934602] LustreError: 180160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7062.200467] Lustre: server umount lustre-MDT0001 complete [ 7077.006811] Lustre: server umount lustre-OST0000 complete [ 7086.409501] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 7097.413919] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 7132.611957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7132.849618] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7132.877708] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7132.957548] Lustre: lustre-MDT0000: new disk, initializing [ 7133.018653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7133.031886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7137.178962] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7146.873959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7146.986523] Lustre: 185520:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7147.043618] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7147.047988] Lustre: Skipped 1 previous similar message [ 7147.116758] Lustre: lustre-MDT0001: new disk, initializing [ 7147.195941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7147.237745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7147.248906] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7151.583857] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7160.988656] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7168.253082] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7175.447683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7175.662801] Lustre: lustre-OST0000: new disk, initializing [ 7175.665996] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7175.670932] Lustre: 187487:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7175.746367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7182.856551] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7183.993427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7184.016341] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7184.182559] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7193.903765] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7199.191822] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7199.604904] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7204.735685] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7205.083395] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7209.969092] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7209.973948] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7209.980465] Lustre: Skipped 1 previous similar message [ 7209.983071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7210.031892] LustreError: Skipped 1 previous similar message [ 7220.193903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7220.205176] Lustre: Skipped 4 previous similar messages [ 7221.216088] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7221.305535] Lustre: server umount lustre-OST0000 complete [ 7225.314720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7225.321881] Lustre: Skipped 1 previous similar message [ 7231.228301] Lustre: server umount lustre-MDT0000 complete [ 7234.735515] LustreError: 185513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786069535 with bad export cookie 16308770956145131928 [ 7234.742635] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7234.748526] LustreError: 185513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7235.020861] Lustre: server umount lustre-MDT0001 complete [ 7244.737424] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 22:25:43 (1786069543) [ 7262.706479] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 7265.565106] Key type lgssc unregistered [ 7265.966763] LNet: 190440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7265.983780] LNetError: 190440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7266.012043] LNet: Removed LNI 192.168.203.102@tcp [ 7266.868473] Key type .llcrypt unregistered [ 7266.870878] Key type ._llcrypt unregistered [ 7286.305218] Key type ._llcrypt registered [ 7286.311492] Key type .llcrypt registered [ 7286.668566] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 7289.086849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7289.105261] alg: No test for adler32 (adler32-zlib) [ 7290.246163] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 7290.592732] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 7292.296204] Key type lgssc registered [ 7293.271725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7301.852453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7301.870989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7303.406191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7307.969696] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7312.999561] LustreError: 191659:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7313.014131] LustreError: 191659:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7318.057657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7318.586238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7324.674891] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7332.801639] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7338.325961] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7345.911594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7346.045844] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7346.048614] LustreError: 193524:0:(llog.c:1696:llog_backup()) MGC192.168.203.102@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7346.054621] Lustre: 193524:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.102@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7346.150608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7353.525674] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7360.940768] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7363.559326] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7363.570931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7363.612310] Lustre: Skipped 2 previous similar messages [ 7363.635281] Lustre: Skipped 2 previous similar messages [ 7364.646074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7364.673819] Lustre: Skipped 2 previous similar messages [ 7369.606184] Lustre: server umount lustre-MDT0000 complete [ 7369.709253] LustreError: 191656:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7369.729136] LustreError: 191656:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7372.817788] LustreError: 191639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786069673 with bad export cookie 9999422506587288992 [ 7372.830332] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7372.835302] LustreError: 191639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7372.896369] LustreError: 192388:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000002190107d x1872829707480192/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7373.117962] Lustre: server umount lustre-MDT0001 complete [ 7386.067491] Lustre: server umount lustre-OST0000 complete [ 7395.852320] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 7405.779657] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 7440.046034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7440.281484] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7440.306180] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7440.377214] Lustre: lustre-MDT0000: new disk, initializing [ 7440.472684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7440.485801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7444.538055] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7454.627738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7454.699517] Lustre: 197049:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7454.731847] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7454.734550] Lustre: Skipped 1 previous similar message [ 7454.738775] LustreError: 197834:0:(llog.c:1696:llog_backup()) MGC192.168.203.102@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7454.744767] LustreError: 197834:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7454.748253] Lustre: 197834:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.102@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7454.766226] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7454.770740] Lustre: Skipped 1 previous similar message [ 7454.821820] Lustre: lustre-MDT0001: new disk, initializing [ 7454.894969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7454.931621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7454.944787] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7459.905855] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7468.721537] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7473.047241] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7479.814785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7479.995547] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7480.001137] Lustre: Skipped 1 previous similar message [ 7480.004775] LustreError: 198994:0:(llog.c:1696:llog_backup()) MGC192.168.203.102@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7480.012860] LustreError: 198994:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7480.017088] Lustre: 198994:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.102@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7480.023636] Lustre: 198994:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7480.037420] Lustre: lustre-OST0000: new disk, initializing [ 7480.043060] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7480.048219] Lustre: 199017:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7480.142735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7485.967405] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7489.550136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7489.555932] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7489.649876] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7494.244701] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7498.559722] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7498.821904] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7503.680852] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7503.859337] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7509.988579] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7509.996252] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7510.004043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7515.109188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7515.113231] Lustre: Skipped 2 previous similar messages [ 7520.229587] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7520.237057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7520.241869] Lustre: Skipped 1 previous similar message [ 7520.343618] Lustre: server umount lustre-OST0000 complete [ 7525.350724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7525.372117] Lustre: Skipped 1 previous similar message [ 7530.469328] LustreError: 197057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7530.480655] LustreError: 197057:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7530.497968] Lustre: server umount lustre-MDT0000 complete [ 7535.057718] LustreError: 197042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786069836 with bad export cookie 9999422506587289713 [ 7535.069731] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7535.078312] LustreError: 197042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7535.475836] Lustre: server umount lustre-MDT0001 complete [ 7544.173493] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 22:30:43 (1786069843) [ 7561.678473] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 7573.112869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7573.626621] LustreError: 202146:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7573.675541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7578.425350] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7579.107072] LustreError: 202147:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7587.982292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7592.583281] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7596.605254] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7603.366539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7611.539474] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7622.010232] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7622.295221] Lustre: lustre-OST0001: new disk, initializing [ 7622.298647] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7622.303358] Lustre: 204453:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7622.603898] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7622.609856] Lustre: Skipped 2 previous similar messages [ 7623.651790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7623.660723] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7623.735962] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7628.648603] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7640.662432] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7651.297078] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7654.174404] Lustre: DEBUG MARKER: ADD OST3 [ 7659.491260] loop: module loaded [ 7659.605716] loop0: detected capacity change from 0 to 204800000 [ 7659.934510] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7659.960548] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7659.979151] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7668.929970] loop0: detected capacity change from 0 to 204800000 [ 7668.980385] Lustre: DEBUG MARKER: STOP OST3 [ 7679.352412] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7680.629855] Lustre: lustre-OST0003: new disk, initializing [ 7680.650798] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7680.669478] Lustre: 206514:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7682.232075] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7682.358195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7682.369108] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7682.418330] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7684.257400] LustreError: 206586:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7685.088380] Lustre: Failing over lustre-OST0003 [ 7685.189544] Lustre: server umount lustre-OST0003 complete [ 7687.436578] LustreError: 203634:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7687.474223] LustreError: 203634:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7687.659564] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7687.666060] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7687.695625] Lustre: Skipped 2 previous similar messages [ 7690.835243] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7704.119374] Lustre: DEBUG MARKER: START OST3 again [ 7708.000470] LustreError: 203634:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7708.037511] LustreError: 203634:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 7711.445590] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7711.798074] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7711.838063] Lustre: Skipped 1 previous similar message [ 7711.870043] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7713.464693] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7713.560396] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7713.562515] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7719.412728] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7741.769595] Lustre: Failing over lustre-OST0003 [ 7741.837708] Lustre: server umount lustre-OST0003 complete [ 7742.467974] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7742.474215] LustreError: 208729:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7742.498219] Lustre: Skipped 1 previous similar message [ 7742.523188] LustreError: 208729:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7762.401797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7762.408541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7762.427690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7762.431477] Lustre: Skipped 2 previous similar messages [ 7765.525932] Lustre: server umount lustre-MDT0000 complete [ 7769.613428] LustreError: 202126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786070070 with bad export cookie 9999422506587290588 [ 7769.618722] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7769.623438] LustreError: 202126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7776.222921] Lustre: server umount lustre-MDT0001 complete [ 7779.939414] Lustre: server umount lustre-OST0000 complete [ 7783.611549] Lustre: server umount lustre-OST0001 complete [ 7790.606936] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 7800.153385] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 7838.923402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7839.253568] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7839.279888] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7839.372844] Lustre: lustre-MDT0000: new disk, initializing [ 7839.455375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7843.357240] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7856.081096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7856.215731] Lustre: 212071:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7856.270850] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7856.275102] Lustre: Skipped 1 previous similar message [ 7856.347877] Lustre: lustre-MDT0001: new disk, initializing [ 7856.505933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7856.511149] Lustre: Skipped 1 previous similar message [ 7856.553356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7856.571301] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7861.312174] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7871.325064] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7878.957355] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7886.442590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7886.660615] Lustre: 214038:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7888.290942] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7888.785757] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7893.340536] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 7903.054137] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7908.716341] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7908.913944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7913.467245] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7913.657338] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7919.075924] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7919.081942] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7919.092036] Lustre: Skipped 5 previous similar messages [ 7919.096509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7919.101816] Lustre: Skipped 7 previous similar messages [ 7921.738292] Lustre: server umount lustre-OST0000 complete [ 7928.289637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7928.296813] LustreError: Skipped 1 previous similar message [ 7932.654239] Lustre: server umount lustre-MDT0000 complete [ 7934.952828] LustreError: 214046:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7934.980932] LustreError: 214046:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 7936.975935] LustreError: 212062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786070237 with bad export cookie 9999422506587292884 [ 7937.004141] LustreError: 212062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7937.010216] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7937.575199] Lustre: server umount lustre-MDT0001 complete [ 7942.421928] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 22:37:20 (1786070240) [ 7961.708242] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 7973.152556] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 8016.374315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8016.573970] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8016.596344] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8016.600578] Lustre: Skipped 1 previous similar message [ 8016.679862] Lustre: lustre-MDT0000: new disk, initializing [ 8016.692779] Lustre: Skipped 1 previous similar message [ 8016.848503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8016.862235] Lustre: Skipped 1 previous similar message [ 8021.371720] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8034.034576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8034.173554] Lustre: 218650:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8034.688052] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8040.016302] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8051.444615] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8058.132606] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8066.166498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8066.642801] Lustre: 220620:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8068.491878] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8068.767629] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8074.529798] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8083.515866] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8089.018261] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8089.395217] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8094.671767] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8095.066460] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8099.305525] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8099.318592] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8099.321057] LustreError: Skipped 1 previous similar message [ 8099.325485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8099.325496] Lustre: Skipped 4 previous similar messages [ 8099.365654] Lustre: Skipped 4 previous similar messages [ 8103.638887] Lustre: server umount lustre-OST0000 complete [ 8123.360366] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8123.485987] Lustre: server umount lustre-MDT0000 complete [ 8124.903547] LustreError: 218663:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8124.931788] LustreError: 218663:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8128.153045] LustreError: 218642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786070429 with bad export cookie 9999422506587293766 [ 8128.162629] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8128.168735] LustreError: 218642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8128.640491] Lustre: server umount lustre-MDT0001 complete [ 8139.652629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8140.468988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8140.473661] Lustre: Skipped 4 previous similar messages [ 8145.522739] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8155.545551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8161.243254] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8169.689250] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8177.328723] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8184.667958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8190.764331] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8199.018209] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8210.915416] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8210.918574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8210.927335] Lustre: Skipped 3 previous similar messages [ 8210.936995] Lustre: Skipped 8 previous similar messages [ 8215.025760] Lustre: server umount lustre-OST0000 complete [ 8224.979856] Lustre: server umount lustre-MDT0000 complete [ 8229.193919] LustreError: 222822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786070530 with bad export cookie 9999422506587294648 [ 8229.195278] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8229.207765] LustreError: 222822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8229.525419] Lustre: server umount lustre-MDT0001 complete [ 8341.596636] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 8344.250260] Key type lgssc unregistered [ 8344.454679] LNet: 226510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8344.462893] LNetError: 226510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8344.474923] LNet: Removed LNI 192.168.203.102@tcp [ 8345.442242] Key type .llcrypt unregistered [ 8345.447805] Key type ._llcrypt unregistered [ 8374.342893] Key type ._llcrypt registered [ 8374.345370] Key type .llcrypt registered [ 8374.441801] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 8390.944672] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 8392.528227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8392.569598] alg: No test for adler32 (adler32-zlib) [ 8393.613796] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 8393.777065] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 8395.400167] Key type lgssc registered [ 8396.879148] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8437.922818] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8437.961579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8439.179642] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8439.219643] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8439.286104] Lustre: lustre-MDT0000: new disk, initializing [ 8439.489824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8439.507481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8444.619776] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8456.085495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8456.298533] Lustre: 229867:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8456.429752] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8456.434128] Lustre: Skipped 1 previous similar message [ 8456.669496] Lustre: lustre-MDT0001: new disk, initializing [ 8456.909870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8456.959445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8457.052509] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8461.905278] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8471.504874] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8478.101252] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8487.016738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8487.321255] Lustre: lustre-OST0000: new disk, initializing [ 8487.324717] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8487.334792] Lustre: 231836:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8487.434223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8494.546214] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8495.319850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8495.334299] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8495.570681] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8504.860201] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8509.507982] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8509.672597] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8514.744929] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8514.903741] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8521.188223] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8521.206045] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8521.209665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8523.158203] Lustre: server umount lustre-OST0000 complete [ 8529.405025] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8529.418512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8529.434421] Lustre: Skipped 1 previous similar message [ 8529.443657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8529.451611] Lustre: Skipped 1 previous similar message [ 8531.425725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8531.428198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8531.441688] Lustre: Skipped 1 previous similar message [ 8533.489685] Lustre: server umount lustre-MDT0000 complete [ 8537.624474] LustreError: 229859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786070838 with bad export cookie 2668666961901095581 [ 8537.626901] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8537.639609] LustreError: 229859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8537.898244] Lustre: server umount lustre-MDT0001 complete [ 8556.027884] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 8568.059084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8568.467846] LustreError: 234770:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8568.495825] LustreError: 234770:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8568.568606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8573.173538] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8573.924630] LustreError: 234771:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8578.022256] LustreError: 234770:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8583.145902] LustreError: 234771:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8583.274304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8583.649724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8588.564552] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8591.722026] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8599.413199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8600.066242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8607.618507] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8616.504136] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8616.630810] Lustre: lustre-OST0001: new disk, initializing [ 8616.634161] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8616.639549] Lustre: 237078:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8616.712780] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8623.840670] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8624.577909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8624.588816] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8624.802054] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8634.800524] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8643.479492] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8658.337101] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 22:49:17 (1786070957) [ 8660.461549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8660.471721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8660.477517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8665.597409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8665.602573] Lustre: Skipped 5 previous similar messages [ 8666.818939] Lustre: server umount lustre-MDT0000 complete [ 8670.310613] LustreError: 237076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786070971 with bad export cookie 2668666961901096463 [ 8670.311466] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8670.387235] LustreError: 237076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8670.666820] Lustre: server umount lustre-MDT0001 complete [ 8684.711680] Lustre: server umount lustre-OST0000 complete [ 8686.052333] Lustre: 228091:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786070971/real 1786070971] req@ffff98715423ed80 x1872830864143616/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786070987 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8686.089569] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8686.113368] Lustre: Skipped 3 previous similar messages [ 8688.440393] Lustre: server umount lustre-OST0001 complete [ 8693.975840] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 8703.565609] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 8737.159629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8737.326873] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8737.344695] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8737.420991] Lustre: lustre-MDT0000: new disk, initializing [ 8737.493786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8737.561899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8742.306517] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8754.115469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8754.241157] Lustre: 241029:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8754.407544] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8754.411325] Lustre: Skipped 1 previous similar message [ 8754.472847] Lustre: lustre-MDT0001: new disk, initializing [ 8754.604596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8754.614825] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8759.520324] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8769.077863] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8775.443727] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8781.386717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8781.595654] Lustre: lustre-OST0000: new disk, initializing [ 8781.608563] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8781.628805] Lustre: 242997:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8783.574555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8783.584957] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8783.654371] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8787.876454] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8797.125111] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8801.747732] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8802.087973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8807.132520] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8807.432133] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8814.567480] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8814.570769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8814.570943] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8814.598053] Lustre: Skipped 1 previous similar message [ 8824.288492] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8824.389789] Lustre: server umount lustre-OST0000 complete [ 8835.041603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8835.050590] Lustre: Skipped 6 previous similar messages [ 8842.208138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8842.406162] Lustre: server umount lustre-MDT0000 complete [ 8844.263322] LustreError: 241034:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8844.279473] LustreError: 241034:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8846.041472] LustreError: 241020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786071147 with bad export cookie 2668666961901098598 [ 8846.042815] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8846.050217] LustreError: 241020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8846.281788] Lustre: server umount lustre-MDT0001 complete [ 8853.659616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8854.001494] LustreError: 245215:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8854.059975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8854.085294] Lustre: Skipped 2 previous similar messages [ 8857.941905] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8866.561820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8870.777161] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8877.552474] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8884.416318] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8890.519168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8897.119463] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 8903.529868] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9050.593341] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9050.604745] LustreError: Skipped 1 previous similar message [ 9050.617248] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9050.651286] Lustre: Skipped 2 previous similar messages [ 9050.660751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9050.672691] Lustre: Skipped 2 previous similar messages [ 9056.439388] Lustre: server umount lustre-OST0000 complete [ 9061.345865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9075.168611] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9075.268138] Lustre: server umount lustre-MDT0000 complete [ 9077.219941] LustreError: 245216:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9077.240885] LustreError: 245216:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9079.223913] LustreError: 245196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786071380 with bad export cookie 2668666961901099480 [ 9079.227185] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9079.231429] LustreError: 245196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9079.634905] Lustre: server umount lustre-MDT0001 complete [ 9171.605286] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 9173.693791] Key type lgssc unregistered [ 9174.157653] LNet: 248828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9174.163996] LNetError: 248828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9175.207298] LNet: Removed LNI 192.168.203.102@tcp [ 9175.988204] Key type .llcrypt unregistered [ 9175.994088] Key type ._llcrypt unregistered [ 9205.716236] Key type ._llcrypt registered [ 9205.718377] Key type .llcrypt registered [ 9205.801616] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 9225.086244] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 9228.082043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9228.106142] alg: No test for adler32 (adler32-zlib) [ 9229.207991] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [ 9229.584820] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 9231.208178] Key type lgssc registered [ 9232.127888] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9274.471699] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 9285.893585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9285.915656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9287.120595] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9287.142216] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9287.193957] Lustre: lustre-MDT0000: new disk, initializing [ 9287.244973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9287.259572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9291.422354] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9301.573191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9301.690864] Lustre: 252901:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9301.733675] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9301.737572] Lustre: Skipped 1 previous similar message [ 9301.798188] Lustre: lustre-MDT0001: new disk, initializing [ 9301.899850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9301.931865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9301.945668] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9306.785247] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9311.891599] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9318.342313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9318.548433] Lustre: lustre-OST0000: new disk, initializing [ 9318.551832] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9318.556485] Lustre: 254534:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9318.645478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9319.867696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9319.874519] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9320.052798] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9324.924367] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9335.730815] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9335.834290] Lustre: lustre-OST0001: new disk, initializing [ 9335.836755] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9335.841188] Lustre: 255405:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9335.918939] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9342.886893] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9343.671951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9343.686383] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9343.726153] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9356.005753] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9364.922555] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9377.426252] Lustre: server umount lustre-MDT0000 complete [ 9378.787568] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9378.791937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9379.815068] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9379.828762] Lustre: Skipped 1 previous similar message [ 9380.945118] LustreError: 256172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786071681 with bad export cookie 3850883512702354320 [ 9380.967504] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9380.969767] LustreError: 256172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9381.342313] Lustre: server umount lustre-MDT0001 complete [ 9395.771461] Lustre: server umount lustre-OST0000 complete [ 9410.406474] Lustre: server umount lustre-OST0001 complete [ 9424.748349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9435.515232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9444.431572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9453.716621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9472.298839] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 9482.320828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9482.463405] Lustre: lustre-MDT0000: reset Object Index mappings [ 9482.810654] LustreError: 260426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9482.824161] LustreError: 260426:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 9482.865945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9486.608931] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9488.359835] LustreError: 260427:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9492.469458] LustreError: 260426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9495.209290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9495.228298] Lustre: lustre-MDT0001: reset Object Index mappings [ 9495.576051] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9500.264890] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9502.702591] Lustre: 261568:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9510.107410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9517.723936] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9520.888065] LustreError: 261923:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9526.167651] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9526.350428] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9526.356508] Lustre: Skipped 1 previous similar message [ 9527.436702] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9527.503360] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9531.978416] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9539.471546] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9543.247687] Lustre: 263442:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9548.259505] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9548.260629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9548.270696] Lustre: Skipped 4 previous similar messages [ 9548.280033] Lustre: Skipped 3 previous similar messages [ 9553.350351] Lustre: server umount lustre-MDT0000 complete [ 9553.394370] LustreError: 260422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9553.420345] LustreError: 260422:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9556.496672] LustreError: 260406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786071857 with bad export cookie 3850883512702361110 [ 9556.503558] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9556.513193] LustreError: 260406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9556.732617] Lustre: server umount lustre-MDT0001 complete [ 9570.935379] Lustre: server umount lustre-OST0000 complete [ 9585.006511] Lustre: server umount lustre-OST0001 complete [ 9606.208288] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 23:05:05 (1786071905) [ 9624.645543] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 9634.027504] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 9671.256282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9671.490478] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9671.520421] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9671.856359] Lustre: lustre-MDT0000: new disk, initializing [ 9671.927476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9671.939541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9676.008866] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9686.648885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9686.852190] Lustre: 267695:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9686.912274] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9686.916171] Lustre: Skipped 1 previous similar message [ 9687.040746] Lustre: lustre-MDT0001: new disk, initializing [ 9687.181389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9687.219796] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9691.874492] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9701.055310] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9707.321472] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9716.341975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9716.569741] Lustre: lustre-OST0000: new disk, initializing [ 9716.573197] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9716.578498] Lustre: 269666:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9718.250049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9718.255207] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9718.405185] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9722.383380] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9732.603664] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9736.998235] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9737.167829] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9741.583845] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9741.756762] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9743.844740] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9743.851420] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9743.879514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9749.479138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9749.492440] Lustre: Skipped 3 previous similar messages [ 9749.874723] Lustre: server umount lustre-OST0000 complete [ 9753.572443] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9753.583561] LustreError: Skipped 1 previous similar message [ 9753.588448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9753.602175] Lustre: Skipped 1 previous similar message [ 9753.610919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9759.548274] Lustre: server umount lustre-MDT0000 complete [ 9759.717745] LustreError: 267706:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9759.739925] LustreError: 267706:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9762.827968] LustreError: 268490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786072063 with bad export cookie 3850883512702363203 [ 9762.847764] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9762.851720] LustreError: 268490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9763.213444] Lustre: server umount lustre-MDT0001 complete [ 9780.962903] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 9791.483238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9791.978627] LustreError: 272598:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9792.061825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9792.068561] Lustre: Skipped 2 previous similar messages [ 9797.017972] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9806.316076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9811.287746] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9814.205741] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9822.676231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9828.942886] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9836.700567] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9836.839708] Lustre: lustre-OST0001: new disk, initializing [ 9836.843327] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9836.848382] Lustre: 274906:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9838.093263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9838.108426] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9838.299427] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9843.589705] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9853.318567] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9860.133431] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9867.817575] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9868.136500] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9879.528244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9879.531733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9879.537076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9879.545986] Lustre: Skipped 4 previous similar messages [ 9879.603674] Lustre: Skipped 5 previous similar messages [ 9881.107375] Lustre: server umount lustre-MDT0000 complete [ 9884.442478] LustreError: 274905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786072185 with bad export cookie 3850883512702364085 [ 9884.449473] LustreError: 274905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9884.461962] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9884.665260] LustreError: 275274:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9884.678731] LustreError: 275274:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9884.869207] Lustre: server umount lustre-MDT0001 complete [ 9891.056972] Lustre: server umount lustre-OST0000 complete [ 9894.867659] Lustre: server umount lustre-OST0001 complete [ 9911.626938] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 9922.190442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9922.489924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9922.493889] Lustre: Skipped 3 previous similar messages [ 9926.923786] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9936.682550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9942.346591] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9945.374094] Lustre: 279106:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9952.604359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9958.685217] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9961.070057] LustreError: 279460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9961.086865] LustreError: 279460:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9967.420820] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9968.700848] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9968.768290] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9974.099554] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [ 9982.808433] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9987.197801] Lustre: 280979:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9994.110420] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9994.332664] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10005.905480] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 23:11:45 (1786072305) [10019.814569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10019.821377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10019.826198] Lustre: Skipped 5 previous similar messages [10019.839482] Lustre: Skipped 5 previous similar messages [10026.069518] Lustre: server umount lustre-MDT0000 complete [10029.965139] LustreError: 277944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786072331 with bad export cookie 3850883512702370679 [10029.971455] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10029.976032] LustreError: 277944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10039.265590] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10039.271850] Lustre: Skipped 9 previous similar messages [10044.384158] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10044.590317] Lustre: server umount lustre-MDT0001 complete [10063.328231] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10063.420753] Lustre: server umount lustre-OST0000 complete [10067.171435] Lustre: server umount lustre-OST0001 complete [10072.969519] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [10083.051670] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [10126.608521] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [10136.326342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10136.813644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10136.845295] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10136.895380] Lustre: lustre-MDT0000: new disk, initializing [10136.966770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10141.882419] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10153.744434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10153.930856] Lustre: 285221:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10154.343306] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10159.470173] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10170.064612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10170.424984] Lustre: 286855:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10172.196069] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10176.905839] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10188.822572] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10189.046402] Lustre: 287727:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10189.162063] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10189.171430] Lustre: Skipped 6 previous similar messages [10190.889041] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10190.895282] Lustre: Skipped 1 previous similar message [10191.031385] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10196.174584] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10206.340861] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10210.761594] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10210.766300] Lustre: Skipped 1 previous similar message [10225.201354] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 23:15:24 (1786072524) [10227.177504] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10227.184224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10227.201178] Lustre: Skipped 3 previous similar messages [10227.243630] Lustre: Skipped 5 previous similar messages [10229.421300] Lustre: server umount lustre-MDT0000 complete [10232.373876] LustreError: 286385:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10232.396227] LustreError: 286385:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [10232.610686] LustreError: 285212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786072533 with bad export cookie 3850883512702378393 [10232.620688] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10232.623644] LustreError: 285212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10232.917773] Lustre: server umount lustre-MDT0001 complete [10246.775794] Lustre: server umount lustre-OST0000 complete [10259.936391] Lustre: server umount lustre-OST0001 complete [10267.810782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10271.746202] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10279.844847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10284.315901] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10291.391314] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10297.211173] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10305.311840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10311.345781] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10319.428171] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10343.491507] loop1: detected capacity change from 0 to 204800000 [10343.701543] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10343.708936] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10343.719038] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10349.134729] loop1: detected capacity change from 0 to 204800000 [10354.007458] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10354.140250] LustreError: 289961:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs17466: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10354.154671] LustreError: 289961:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs17466 [10354.158297] LustreError: 289961:0:(mgs_handler.c:517:mgs_target_reg()) fs17466-MDT0001: Error checking target index: rc = -13 [10354.164729] LustreError: fs17466-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10354.174524] LustreError: 293714:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10354.179323] LustreError: 293714:0:(tgt_mount.c:2129:server_put_super()) no obd fs17466-MDT0001 [10354.187357] LustreError: 293714:0:(tgt_mount.c:116:server_deregister_mount()) fs17466-MDT0001 not registered [10354.228886] Lustre: server umount fs17466-MDT0001 complete [10354.231173] LustreError: 293714:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [10362.077494] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10362.146311] Lustre: Setting parameter fs17466-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs17466-MDT0001 [10362.227718] Lustre: srv-fs17466-MDT0001: No data found on store. Initialize space. [10362.239303] Lustre: Skipped 4 previous similar messages [10362.316551] Lustre: fs17466-MDT0001: new disk, initializing [10362.319672] Lustre: Skipped 3 previous similar messages [10362.460909] Lustre: srv-fs17466-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10362.470549] Lustre: 294180:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs17466-MDT0001-mdtlov: can't create new llog: rc = -115 [10362.487810] LustreError: 294180:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs17466-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10367.113932] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10374.996497] Lustre: server umount fs17466-MDT0001 complete [10387.926367] loop1: detected capacity change from 0 to 204800000 [10392.558912] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10393.162393] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10393.554624] Lustre: 295747:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs17466-MDT0001-mdtlov: can't create new llog: rc = -115 [10393.567298] LustreError: 295747:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs17466-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10398.631866] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10402.927510] Lustre: server umount fs17466-MDT0001 complete [10407.921705] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10412.284913] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.2@tcp (stopping) [10413.027839] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10413.052215] Lustre: Skipped 2 previous similar messages [10418.429162] Lustre: server umount lustre-MDT0000 complete [10425.831627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10425.905429] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10430.540734] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10431.490416] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10438.839478] loop1: detected capacity change from 0 to 204800000 [10444.236538] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10444.495350] Lustre: 298192:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs17466-MDT0001-mdtlov: can't create new llog: rc = -115 [10444.506359] LustreError: 298192:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs17466-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10448.271439] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10452.500978] Lustre: server umount fs17466-MDT0001 complete [10472.427789] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10472.436867] LustreError: Skipped 1 previous similar message [10478.505984] Lustre: server umount lustre-OST0000 complete [10487.982230] Lustre: server umount lustre-MDT0000 complete [10491.589632] LustreError: 290714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786072792 with bad export cookie 3850883512702381074 [10491.595274] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10491.597985] LustreError: 290714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10491.875405] Lustre: server umount lustre-MDT0001 complete [10501.641906] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [10503.752621] Key type lgssc unregistered [10503.958717] LNet: 300539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10503.963960] LNetError: 300539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10503.989587] LNet: Removed LNI 192.168.203.102@tcp [10504.943167] Key type .llcrypt unregistered [10504.944855] Key type ._llcrypt unregistered [10527.663621] Key type ._llcrypt registered [10527.665716] Key type .llcrypt registered [10527.738315] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [10529.361407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10530.538485] alg: No test for adler32 (adler32-zlib) [10532.327683] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [10532.526685] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [10534.168295] Key type lgssc registered [10536.772660] Lustre: Echo OBD driver; http://www.lustre.org/ [10547.282415] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10547.293316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10548.730518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10553.872592] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10558.952900] LustreError: 301812:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10558.962551] LustreError: 301812:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10563.566584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10564.208969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10569.244547] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10576.254657] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10583.548722] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10591.599940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10591.864130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10593.899814] LustreError: 303645:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10598.182821] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10599.421059] LustreError: 303646:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10604.527458] LustreError: 303785:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10605.206391] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10608.674657] LustreError: 303785:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10608.717740] LustreError: 303785:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10618.087739] LustreError: 303645:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10618.131113] LustreError: 303645:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10626.431303] loop1: detected capacity change from 0 to 204800000 [10626.638478] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10626.649247] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10626.657984] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10632.350796] loop1: detected capacity change from 0 to 204800000 [10635.266650] LustreError: 303785:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10635.327358] LustreError: 303785:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [10639.370234] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10639.480717] Lustre: fs30578-OST0000: new disk, initializing [10639.487227] Lustre: srv-fs30578-OST0000: No data found on store. Initialize space. [10639.495876] Lustre: 305551:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10639.572474] Lustre: fs30578-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10646.122410] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10654.872584] Lustre: server umount fs30578-OST0000 complete [10667.653469] loop1: detected capacity change from 0 to 204800000 [10669.789082] LustreError: 304317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10669.806710] LustreError: 304317:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [10671.075110] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10673.333763] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10673.474382] Lustre: fs30578-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10679.980954] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10691.577197] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10692.113694] loop2: detected capacity change from 0 to 204800000 [10692.625148] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10692.648828] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10692.669195] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10698.094913] loop2: detected capacity change from 0 to 204800000 [10704.037295] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10704.083824] LustreError: fs30578-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10704.093128] LustreError: fs30578-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10704.098886] LustreError: 308590:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10704.112472] LustreError: 308590:0:(tgt_mount.c:2129:server_put_super()) no obd fs30578-OST0001 [10704.118079] LustreError: 308590:0:(tgt_mount.c:116:server_deregister_mount()) fs30578-OST0001 not registered [10704.147246] Lustre: server umount fs30578-OST0001 complete [10704.149876] LustreError: 308590:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [10716.276505] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10716.411193] Lustre: fs30578-OST0001: new disk, initializing [10716.413980] Lustre: srv-fs30578-OST0001: No data found on store. Initialize space. [10716.418875] Lustre: 309143:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10716.488385] Lustre: fs30578-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10722.632524] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10731.267854] Lustre: server umount fs30578-OST0000 complete [10736.346156] LustreError: 303643:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10736.377262] LustreError: 303643:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [10742.308193] Lustre: server umount fs30578-OST0001 complete [10755.750460] loop1: detected capacity change from 0 to 204800000 [10761.047369] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10761.476468] Lustre: fs30578-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10768.894637] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10776.663677] loop2: detected capacity change from 0 to 204800000 [10782.791949] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10789.210912] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10799.384756] Lustre: server umount fs30578-OST0000 complete [10810.308813] Lustre: server umount fs30578-OST0001 complete [10823.685299] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10825.699143] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10825.703857] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10825.711869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10828.780304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10828.786716] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10829.945474] Lustre: server umount lustre-OST0000 complete [10833.889779] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10833.893127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10833.896969] Lustre: Skipped 1 previous similar message [10833.903820] Lustre: Skipped 2 previous similar messages [10839.009840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10839.020044] Lustre: Skipped 1 previous similar message [10844.149863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10844.182468] Lustre: Skipped 1 previous similar message [10848.224269] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10848.512870] Lustre: server umount lustre-MDT0000 complete [10852.572871] LustreError: 302556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786073153 with bad export cookie 9686877250159688415 [10852.574630] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10852.603942] LustreError: 302556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10852.814828] Lustre: server umount lustre-MDT0001 complete [10865.646413] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [10868.133462] Key type lgssc unregistered [10868.356819] LNet: 315071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10868.361854] LNetError: 315071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10868.376327] LNet: Removed LNI 192.168.203.102@tcp [10869.093200] Key type .llcrypt unregistered [10869.098108] Key type ._llcrypt unregistered [10909.329838] Key type ._llcrypt registered [10909.332826] Key type .llcrypt registered [10909.405708] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [10925.478972] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [10929.328659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10929.361066] alg: No test for adler32 (adler32-zlib) [10930.720793] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [10930.954521] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [10932.656182] Key type lgssc registered [10933.500618] Lustre: Echo OBD driver; http://www.lustre.org/ [10972.413698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10972.450091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10973.783081] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10973.826394] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10973.942950] Lustre: lustre-MDT0000: new disk, initializing [10974.013260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10974.031861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10978.736799] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [10989.405542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10989.505487] Lustre: 318936:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10989.545547] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10989.548898] Lustre: Skipped 1 previous similar message [10989.619394] Lustre: lustre-MDT0001: new disk, initializing [10989.752665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10989.792213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10989.800857] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10993.974407] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [11004.173959] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11009.406226] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11016.438355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11016.760381] Lustre: lustre-OST0000: new disk, initializing [11016.763889] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11016.773606] Lustre: 320904:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11016.869868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11023.427752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11023.441318] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11023.614303] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11024.148890] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [11033.561558] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11037.013801] Lustre: Mounted lustre-client - version 2.17.56_50_gf66e916 [11038.278820] Lustre: Unmounted lustre-client [11047.397116] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11047.398501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11047.426495] Lustre: Skipped 1 previous similar message [11047.438567] Lustre: Skipped 1 previous similar message [11052.514858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11052.520058] Lustre: Skipped 1 previous similar message [11053.223315] Lustre: server umount lustre-OST0000 complete [11057.633179] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11057.635223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11057.640552] Lustre: Skipped 1 previous similar message [11057.663531] Lustre: Skipped 1 previous similar message [11062.754690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11062.758746] Lustre: Skipped 1 previous similar message [11063.506973] Lustre: server umount lustre-MDT0000 complete [11066.952790] LustreError: 318930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786073367 with bad export cookie 17206250515263782069 [11066.957182] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11066.993181] LustreError: 318930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11067.442220] Lustre: server umount lustre-MDT0001 complete [11080.859678] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [11084.050279] Key type lgssc unregistered [11086.432619] LNet: 323070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11086.440753] LNetError: 323070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11166.759398] LNet: Removed LNI 192.168.203.102@tcp [11167.338385] Key type .llcrypt unregistered [11167.348259] Key type ._llcrypt unregistered [11185.233725] Key type ._llcrypt registered [11185.236217] Key type .llcrypt registered [11185.420968] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [11201.554421] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [11202.856168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11202.880242] alg: No test for adler32 (adler32-zlib) [11204.304571] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [11204.574083] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [11206.384171] Key type lgssc registered [11207.818630] Lustre: Echo OBD driver; http://www.lustre.org/ [11238.559397] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 23:32:17 (1786073537) [11244.966727] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11245.010794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11246.338620] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11246.409843] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11246.459284] Lustre: lustre-MDT0000: new disk, initializing [11246.507494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11246.519274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11249.730612] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [11260.450380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11260.552177] Lustre: 326293:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11260.606914] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11260.610942] Lustre: Skipped 1 previous similar message [11260.670861] Lustre: lustre-MDT0001: new disk, initializing [11260.789522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11260.859758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11260.882785] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11264.680489] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [11273.327851] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11278.336079] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11285.541675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11285.704812] Lustre: lustre-OST0000: new disk, initializing [11285.708155] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11285.712990] Lustre: 328262:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11285.811292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11292.284952] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [11292.810675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11292.837538] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11293.009175] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11301.175954] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11318.758777] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11318.766122] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11318.784547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11318.793439] Lustre: Skipped 1 previous similar message [11321.833396] Lustre: server umount lustre-OST0000 complete [11327.457137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11327.465086] LustreError: Skipped 1 previous similar message [11327.467885] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11327.475703] Lustre: Skipped 1 previous similar message [11327.479053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11328.993496] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11328.994761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11329.012200] Lustre: Skipped 1 previous similar message [11334.125545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11338.211153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11338.217904] Lustre: Skipped 2 previous similar messages [11339.232118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11339.370957] Lustre: server umount lustre-MDT0000 complete [11342.383707] LustreError: 327087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786073643 with bad export cookie 1794985538460782686 [11342.387955] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11342.395817] LustreError: 327087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11342.765658] Lustre: server umount lustre-MDT0001 complete [11352.705321] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [11355.075169] Key type lgssc unregistered [11355.289203] LNet: 330174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11355.306533] LNetError: 330174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11355.320443] LNet: Removed LNI 192.168.203.102@tcp [11355.908526] Key type .llcrypt unregistered [11355.912982] Key type ._llcrypt unregistered [11375.705765] Key type ._llcrypt registered [11375.709377] Key type .llcrypt registered [11375.796943] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [11377.122744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11377.205956] alg: No test for adler32 (adler32-zlib) [11378.180715] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [11378.351741] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [11380.080176] Key type lgssc registered [11381.053600] Lustre: Echo OBD driver; http://www.lustre.org/ [11387.555302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11387.566088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11388.922715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11392.686779] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [11398.128723] LustreError: 331448:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11398.154638] LustreError: 331448:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11401.499053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11401.734901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11405.735991] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [11412.159664] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11417.335669] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11423.010647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11423.273542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11428.425849] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [11435.835398] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11443.929183] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11446.650363] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11452.661714] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.2@tcp (stopping) [11452.901971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11452.907991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11453.927629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11453.928923] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11453.933195] Lustre: Skipped 1 previous similar message [11453.958164] Lustre: Skipped 1 previous similar message [11457.025474] Lustre: server umount lustre-MDT0000 complete [11457.772783] LustreError: 331443:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11459.045547] LustreError: 331449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11459.063675] LustreError: 331449:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11460.292630] LustreError: 331427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786073761 with bad export cookie 8263103186819060777 [11460.306616] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11460.310983] LustreError: 331427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11462.881557] LustreError: 331445:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11462.909350] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.2@tcp (stopping) [11464.161695] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11465.185543] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11465.196242] Lustre: Skipped 1 previous similar message [11466.672081] Lustre: server umount lustre-MDT0001 complete [11470.290453] Lustre: server umount lustre-OST0000 complete [11478.999892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11484.071502] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [11486.601324] Lustre: 335629:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.102@tcp [11488.863914] Lustre: server umount MGS complete [11497.531465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11497.960915] LustreError: 336201:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11497.992423] LustreError: 336201:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11498.044627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11503.021188] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [11507.926603] LustreError: 336198:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11507.954729] LustreError: 336198:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11511.027356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11511.256296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11514.925987] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [11523.305801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11523.647791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11530.742339] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [11536.923741] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11538.465448] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11544.962717] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11546.639271] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11552.240671] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11552.248454] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11552.262702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11552.272241] Lustre: Skipped 1 previous similar message [11552.951132] Lustre: server umount lustre-OST0000 complete [11557.357916] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11557.368365] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11557.469127] Lustre: Skipped 2 previous similar messages [11562.465471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11562.469545] Lustre: Skipped 3 previous similar messages [11563.114376] Lustre: server umount lustre-MDT0000 complete [11567.547723] LustreError: 336182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786073868 with bad export cookie 8263103186819061617 [11567.550493] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11567.558627] LustreError: 336182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11567.923543] Lustre: server umount lustre-MDT0001 complete [11581.431662] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [11583.561465] Key type lgssc unregistered [11584.198393] LNet: 339641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11584.207059] LNetError: 339641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11584.230552] LNet: Removed LNI 192.168.203.102@tcp [11585.052749] Key type .llcrypt unregistered [11585.054760] Key type ._llcrypt unregistered [11605.726895] Key type ._llcrypt registered [11605.728953] Key type .llcrypt registered [11605.809932] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [11620.194356] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [11621.446518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11621.464133] alg: No test for adler32 (adler32-zlib) [11622.477125] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [11622.653297] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [11624.320433] Key type lgssc registered [11626.034810] Lustre: Echo OBD driver; http://www.lustre.org/ [11661.936264] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11661.960395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11663.305972] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11663.331233] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11663.579220] Lustre: lustre-MDT0000: new disk, initializing [11663.727557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11663.747110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11668.473159] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [11679.180596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11679.253706] Lustre: 342803:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11679.351274] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11679.354960] Lustre: Skipped 1 previous similar message [11679.406166] Lustre: lustre-MDT0001: new disk, initializing [11679.484762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11679.532895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11679.551086] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11684.510716] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [11694.747253] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11699.879926] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11708.262375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11708.502347] Lustre: lustre-OST0000: new disk, initializing [11708.505890] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11708.512384] Lustre: 344772:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11708.604408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11715.862480] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [11717.167329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11717.200176] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11717.460339] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11726.630760] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11731.217361] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11731.475930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11736.010199] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11736.468166] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11743.201605] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11743.206664] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11743.213330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11743.214319] Lustre: Skipped 1 previous similar message [11743.223229] Lustre: Skipped 1 previous similar message [11744.338927] Lustre: server umount lustre-OST0000 complete [11748.321386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11748.419065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11748.445670] Lustre: Skipped 1 previous similar message [11753.442584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11754.506060] Lustre: server umount lustre-MDT0000 complete [11758.439553] LustreError: 342797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786074059 with bad export cookie 2327774022332737032 [11758.445668] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11758.463502] LustreError: 342797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11758.889724] Lustre: server umount lustre-MDT0001 complete [11770.342768] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 23:41:08 (1786074068) [11786.079981] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [11795.842537] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [11828.300606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11828.434582] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11828.458960] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11828.515562] Lustre: lustre-MDT0000: new disk, initializing [11828.579835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11828.606339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11832.425192] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [11842.669888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11842.771328] Lustre: 349531:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11842.812279] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11842.815974] Lustre: Skipped 1 previous similar message [11842.866331] Lustre: lustre-MDT0001: new disk, initializing [11842.931815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11842.961368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11842.973726] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11847.409094] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [11856.000644] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11862.971614] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11869.400924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11869.674709] Lustre: lustre-OST0000: new disk, initializing [11869.680316] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11869.687688] Lustre: 351499:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11869.787924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11875.971936] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [11876.685528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11876.698037] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11876.916662] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11885.620106] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11917.800309] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11917.802676] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11917.804321] LustreError: Skipped 1 previous similar message [11917.805213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11917.815743] Lustre: Skipped 2 previous similar messages [11917.826301] Lustre: Skipped 2 previous similar messages [11919.931749] Lustre: server umount lustre-OST0000 complete [11923.702770] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11923.717114] LustreError: Skipped 1 previous similar message [11923.728476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11923.768775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11928.034293] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11930.034866] Lustre: server umount lustre-MDT0000 complete [11933.894552] LustreError: 351497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786074234 with bad export cookie 2327774022332737907 [11933.898295] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11933.905290] LustreError: 351497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11934.371874] Lustre: server umount lustre-MDT0001 complete [11944.264057] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [11946.654714] Key type lgssc unregistered [11946.915072] LNet: 353654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11946.921784] LNetError: 353654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11946.944485] LNet: Removed LNI 192.168.203.102@tcp [11947.989499] Key type .llcrypt unregistered [11947.992270] Key type ._llcrypt unregistered [12034.415863] Key type ._llcrypt registered [12034.418234] Key type .llcrypt registered [12034.542832] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [12034.941671] Key type .llcrypt unregistered [12034.946069] Key type ._llcrypt unregistered [12102.237987] Key type ._llcrypt registered [12102.241776] Key type .llcrypt registered [12102.408677] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [12102.684636] Key type .llcrypt unregistered [12102.687181] Key type ._llcrypt unregistered [12132.387183] Key type ._llcrypt registered [12132.390634] Key type .llcrypt registered [12132.627595] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [12132.994697] Key type .llcrypt unregistered [12133.001573] Key type ._llcrypt unregistered [12160.717534] Key type ._llcrypt registered [12160.721276] Key type .llcrypt registered [12160.839741] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [12161.138980] Key type .llcrypt unregistered [12161.147493] Key type ._llcrypt unregistered [12231.219548] Key type ._llcrypt registered [12231.221824] Key type .llcrypt registered [12231.308915] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [12231.564483] Key type .llcrypt unregistered [12231.566270] Key type ._llcrypt unregistered [12260.803625] Key type ._llcrypt registered [12260.805888] Key type .llcrypt registered [12260.887285] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [12261.157400] Key type .llcrypt unregistered [12261.159286] Key type ._llcrypt unregistered [12291.384292] Key type ._llcrypt registered [12291.386702] Key type .llcrypt registered [12291.499802] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [12291.758669] Key type .llcrypt unregistered [12291.760805] Key type ._llcrypt unregistered [12319.158597] Key type ._llcrypt registered [12319.160717] Key type .llcrypt registered [12319.293911] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [12319.622445] Key type .llcrypt unregistered [12319.625787] Key type ._llcrypt unregistered [12344.453328] Key type ._llcrypt registered [12344.455256] Key type .llcrypt registered [12344.533246] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [12344.813795] Key type .llcrypt unregistered [12344.818527] Key type ._llcrypt unregistered [12370.977767] Key type ._llcrypt registered [12370.983880] Key type .llcrypt registered [12371.235457] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [12371.609460] Key type .llcrypt unregistered [12371.611919] Key type ._llcrypt unregistered [12428.551877] Key type ._llcrypt registered [12428.554238] Key type .llcrypt registered [12428.725828] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [12429.482720] Key type .llcrypt unregistered [12429.486638] Key type ._llcrypt unregistered [12457.774622] Key type ._llcrypt registered [12457.777136] Key type .llcrypt registered [12457.919531] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [12458.406457] Key type .llcrypt unregistered [12458.413858] Key type ._llcrypt unregistered [12483.609849] Key type ._llcrypt registered [12483.613239] Key type .llcrypt registered [12483.707380] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [12483.947491] Key type .llcrypt unregistered [12483.949559] Key type ._llcrypt unregistered [12535.701197] Key type ._llcrypt registered [12535.706450] Key type .llcrypt registered [12535.800044] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [12550.425817] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [12553.733696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12553.806978] alg: No test for adler32 (adler32-zlib) [12554.876400] Lustre: Lustre: Build Version: 2.17.56_50_gf66e916 [12555.265600] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [12557.000642] Key type lgssc registered [12558.181530] Lustre: Echo OBD driver; http://www.lustre.org/ [12592.651422] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12592.697536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12593.963162] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12593.983791] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12594.053749] Lustre: tst250fs-MDT0000: new disk, initializing [12594.105960] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12594.125425] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12599.176477] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [12609.645241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12609.749506] Lustre: 373841:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12609.799540] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12609.804449] Lustre: Skipped 1 previous similar message [12609.870397] Lustre: tst250fs-MDT0001: new disk, initializing [12609.950616] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12610.009234] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12610.028850] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12613.769584] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [12618.339683] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12625.579940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12625.944913] Lustre: tst250fs-OST0000: new disk, initializing [12625.964526] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12625.994824] Lustre: 375471:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12626.087922] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12627.706205] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12627.727158] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12627.989343] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12631.742413] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [12641.466501] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12641.615891] Lustre: tst250fs-OST0001: new disk, initializing [12641.621684] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12641.636848] Lustre: 376341:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12641.751719] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12648.065266] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug -1 all [12649.080243] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12649.136358] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12649.465973] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12653.511816] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12655.795623] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [12655.799437] Lustre: Skipped 2 previous similar messages [12675.046381] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12675.054087] Lustre: Skipped 1 previous similar message [12675.054587] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12675.061932] Lustre: Skipped 3 previous similar messages [12679.583325] Lustre: server umount tst250fs-MDT0000 complete [12683.135386] LustreError: 373832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786074984 with bad export cookie 10266499565177792266 [12683.137947] LustreError: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12683.147930] LustreError: 373832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12683.419339] Lustre: server umount tst250fs-MDT0001 complete [12697.896041] Lustre: server umount tst250fs-OST0000 complete [12701.665232] Lustre: 372063:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786074986/real 1786074986] req@ffff987248c92a00 x1872835228271616/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786075002 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12701.695387] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12701.709275] Lustre: Skipped 2 previous similar messages [12701.926751] Lustre: server umount tst250fs-OST0001 complete [12720.133064] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [12727.985390] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [12757.900467] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 23:57:37 (1786075057) [12759.657448] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [12776.153875] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [12786.578285] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [12818.589062] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12507 sec ========= 23:58:37 (1786075117) [12820.591983] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:58:39 (1786075119) === [12823.766392] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:58:42 (1786075122) === [12859.866534] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [12862.219763] Key type lgssc unregistered [12862.484970] LNet: 385483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12862.497824] LNetError: 385483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12862.561975] LNet: Removed LNI 192.168.203.102@tcp [12863.119155] Key type .llcrypt unregistered [12863.123425] Key type ._llcrypt unregistered