[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 1305193170 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002011] Switched APIC routing to physical x2apic. [ 0.003026] kvm-guest: setup PV IPIs [ 0.006000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007014] pid_max: default: 32768 minimum: 301 [ 0.008148] LSM: Security Framework initializing [ 0.009077] Yama: becoming mindful. [ 0.011050] SELinux: Initializing. [ 0.013092] *** VALIDATE selinux *** [ 0.027751] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.034000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.034171] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.035116] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.036162] *** VALIDATE tmpfs *** [ 0.037791] *** VALIDATE proc *** [ 0.039655] *** VALIDATE cgroup *** [ 0.040018] *** VALIDATE cgroup2 *** [ 0.042004] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.044145] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.045013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.046033] Spectre V2 : User space: Vulnerable [ 0.047000] Speculative Store Bypass: Vulnerable [ 0.049218] debug: unmapping init [mem 0xffffffff92e59000-0xffffffff92e60fff] [ 0.051000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.053725] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.054026] ... version: 2 [ 0.055015] ... bit width: 48 [ 0.056020] ... generic registers: 4 [ 0.057015] ... value mask: 0000ffffffffffff [ 0.058014] ... max period: 00007fffffffffff [ 0.059021] ... fixed-purpose events: 3 [ 0.060017] ... event mask: 000000070000000f [ 0.062258] rcu: Hierarchical SRCU implementation. [ 0.065133] smp: Bringing up secondary CPUs ... [ 0.067657] x86: Booting SMP configuration: [ 0.069036] .... node #0, CPUs: #1 #2 #3 [ 0.099110] smp: Brought up 1 node, 4 CPUs [ 0.101024] smpboot: Max logical packages: 1 [ 0.102021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.167021] node 0 deferred pages initialised in 63ms [ 0.178020] devtmpfs: initialized [ 0.179000] x86/mm: Memory block size: 128MB [ 0.184390] gcov: version magic: 0x41383552 [ 0.191249] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.192162] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.193809] pinctrl core: initialized pinctrl subsystem [ 0.197263] [ 0.198010] ************************************************************* [ 0.202017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.210014] ** ** [ 0.218018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.224031] ** ** [ 0.227015] ** This means that this kernel is built to expose internal ** [ 0.228000] ** IOMMU data structures, which may compromise security on ** [ 0.235013] ** your system. ** [ 0.242013] ** ** [ 0.249017] ** If you see this message and you are not debugging the ** [ 0.251017] ** kernel, report this immediately to your vendor! ** [ 0.259013] ** ** [ 0.268016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.270014] ************************************************************* [ 0.287554] NET: Registered protocol family 16 [ 0.291467] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.301086] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.306066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.311098] cpuidle: using governor menu [ 0.318218] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.328864] PCI: Using configuration type 1 for base access [ 0.333140] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.348159] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.349000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.357027] cryptd: max_cpu_qlen set to 1000 [ 0.359216] ACPI: Added _OSI(Module Device) [ 0.360000] ACPI: Added _OSI(Processor Device) [ 0.360000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.364018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.373905] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.386586] ACPI: Interpreter enabled [ 0.387000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.390031] ACPI: Using IOAPIC for interrupt routing [ 0.393170] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.400463] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.422703] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.429051] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.436023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.438081] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.446974] acpiphp: Slot [2] registered [ 0.452183] acpiphp: Slot [5] registered [ 0.455159] acpiphp: Slot [6] registered [ 0.460166] acpiphp: Slot [7] registered [ 0.466789] acpiphp: Slot [8] registered [ 0.473000] acpiphp: Slot [9] registered [ 0.478240] acpiphp: Slot [10] registered [ 0.483156] acpiphp: Slot [3] registered [ 0.488138] acpiphp: Slot [4] registered [ 0.491135] acpiphp: Slot [11] registered [ 0.495152] acpiphp: Slot [12] registered [ 0.498144] acpiphp: Slot [13] registered [ 0.501226] acpiphp: Slot [14] registered [ 0.507023] acpiphp: Slot [15] registered [ 0.512403] acpiphp: Slot [16] registered [ 0.514142] acpiphp: Slot [17] registered [ 0.522149] acpiphp: Slot [18] registered [ 0.526137] acpiphp: Slot [19] registered [ 0.532146] acpiphp: Slot [20] registered [ 0.535151] acpiphp: Slot [21] registered [ 0.539253] acpiphp: Slot [22] registered [ 0.542200] acpiphp: Slot [23] registered [ 0.545135] acpiphp: Slot [24] registered [ 0.549202] acpiphp: Slot [25] registered [ 0.552196] acpiphp: Slot [26] registered [ 0.554482] acpiphp: Slot [27] registered [ 0.560139] acpiphp: Slot [28] registered [ 0.567032] acpiphp: Slot [29] registered [ 0.571132] acpiphp: Slot [30] registered [ 0.572000] acpiphp: Slot [31] registered [ 0.576121] PCI host bridge to bus 0000:00 [ 0.580029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.582030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.588027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.597027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.606016] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.607000] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.612190] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.617190] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.622847] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.635956] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.646088] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.651020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.655023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.658021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.665595] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.677817] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.685172] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.695425] pci 0000:00:01.3: quirk_piix4_acpi+0x0/0x1e0 took 18554 usecs [ 0.704523] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.723016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.749018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.767000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.779220] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.801017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.813019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.844017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.864217] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.890037] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.910018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.938018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.957463] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.967018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.976019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.994022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 1.000655] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 1.007017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 1.013089] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 1.026017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 1.041410] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.048016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.059017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.085015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.096850] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.106017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.114019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.128022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.147232] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.151349] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.154780] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.156399] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.159508] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.166281] iommu: Default domain type: Passthrough [ 1.171139] SCSI subsystem initialized [ 1.174204] ACPI: bus type USB registered [ 1.178117] usbcore: registered new interface driver usbfs [ 1.182173] usbcore: registered new interface driver hub [ 1.185094] usbcore: registered new device driver usb [ 1.187103] pps_core: LinuxPPS API ver. 1 registered [ 1.189014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.193066] PTP clock support registered [ 1.195171] EDAC MC: Ver: 3.0.0 [ 1.197681] PCI: Using ACPI for IRQ routing [ 1.198000] NetLabel: Initializing [ 1.201020] NetLabel: domain hash size = 128 [ 1.203147] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.207135] NetLabel: unlabeled traffic allowed by default [ 1.211117] vgaarb: loaded [ 1.213064] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.216014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.222431] clocksource: Switched to clocksource kvm-clock [ 1.485462] VFS: Disk quotas dquot_6.6.0 [ 1.489364] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.494497] *** VALIDATE ramfs *** [ 1.497388] *** VALIDATE hugetlbfs *** [ 1.499126] pnp: PnP ACPI init [ 1.501666] pnp: PnP ACPI: found 6 devices [ 1.522745] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.526594] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.528909] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.531185] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.533674] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.536179] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.541386] NET: Registered protocol family 2 [ 1.544070] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.552496] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.565147] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.586413] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.601239] TCP: Hash tables configured (established 65536 bind 65536) [ 1.608862] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.613723] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.622186] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.626629] NET: Registered protocol family 1 [ 1.646709] RPC: Registered named UNIX socket transport module. [ 1.658979] RPC: Registered udp transport module. [ 1.666599] RPC: Registered tcp transport module. [ 1.673756] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.680480] NET: Registered protocol family 44 [ 1.690869] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.697426] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.702477] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.710073] PCI: CLS 0 bytes, default 64 [ 1.714392] Unpacking initramfs... [ 5.392214] debug: unmapping init [mem 0xffff90ca3cc54000-0xffff90ca3ffbffff] [ 5.398390] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 5.401657] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 5.406826] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 7.120681] Initialise system trusted keyrings [ 7.124156] Key type blacklist registered [ 7.129669] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 7.150881] zbud: loaded [ 7.163266] *** VALIDATE nfs *** [ 7.164687] *** VALIDATE nfs4 *** [ 7.174183] pstore: using deflate compression [ 7.183659] Platform Keyring initialized [ 7.457439] NET: Registered protocol family 38 [ 7.464722] Key type asymmetric registered [ 7.466947] Asymmetric key parser 'x509' registered [ 7.471024] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 7.476737] io scheduler mq-deadline registered [ 7.478725] io scheduler kyber registered [ 7.482669] io scheduler bfq registered [ 7.488068] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 7.493878] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 7.498728] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 7.509993] ACPI: Power Button [PWRF] [ 7.520109] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 7.539939] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 7.583507] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 7.598794] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 7.631059] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 7.669950] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 7.723244] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 7.734834] Non-volatile memory driver v1.3 [ 7.737360] Linux agpgart interface v0.103 [ 7.884564] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 7.892388] vda: detected capacity change from 0 to 74702848 [ 7.934032] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.942273] vdb: detected capacity change from 0 to 1073741824 [ 7.980726] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.983836] vdc: detected capacity change from 0 to 2621440000 [ 8.011191] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.019298] vdd: detected capacity change from 0 to 2621440000 [ 8.037862] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.040378] vde: detected capacity change from 0 to 4294967296 [ 8.062876] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.065215] vdf: detected capacity change from 0 to 4294967296 [ 8.070040] libphy: Fixed MDIO Bus: probed [ 8.079607] usbcore: registered new interface driver usbserial_generic [ 8.083386] usbserial: USB Serial support registered for generic [ 8.086050] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 8.095505] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 8.113975] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 8.122804] mousedev: PS/2 mouse device common for all mice [ 8.126162] rtc_cmos 00:05: RTC can wake from S4 [ 8.136808] rtc_cmos 00:05: registered as rtc0 [ 8.138667] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 8.155984] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 8.165697] intel_pstate: CPU model not supported [ 8.195121] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 8.206072] hid: raw HID events driver (C) Jiri Kosina [ 8.220102] usbcore: registered new interface driver usbhid [ 8.222159] usbhid: USB HID core driver [ 8.227686] drop_monitor: Initializing network drop monitor service [ 8.232143] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 8.240899] Initializing XFRM netlink socket [ 8.247742] NET: Registered protocol family 10 [ 8.255512] Segment Routing with IPv6 [ 8.257602] NET: Registered protocol family 17 [ 8.265364] mpls_gso: MPLS GSO support [ 8.278751] RAS: Correctable Errors collector initialized. [ 8.281494] AVX version of gcm_enc/dec engaged. [ 8.293173] AES CTR mode by8 optimization enabled [ 8.482819] sched_clock: Marking stable (8482795859, 0)->(11380067918, -2897272059) [ 8.487547] registered taskstats version 1 [ 8.489773] Loading compiled-in X.509 certificates [ 8.492246] zswap: loaded using pool lzo/zbud [ 8.533454] Key type big_key registered [ 8.548691] Key type encrypted registered [ 8.553888] ima: No TPM chip found, activating TPM-bypass! [ 8.556489] ima: Allocated hash algorithm: sha1 [ 8.559278] ima: No architecture policies found [ 8.564550] evm: Initialising EVM extended attributes: [ 8.566688] evm: security.selinux [ 8.568129] evm: security.ima [ 8.569393] evm: security.capability [ 8.571181] evm: HMAC attrs: 0x1 [ 8.573907] rtc_cmos 00:05: setting system clock to 2026-08-13 13:18:33 UTC (1786627113) [ 8.582649] debug: unmapping init [mem 0xffffffff93e03000-0xffffffff93ffffff] [ 8.586343] debug: unmapping init [mem 0xffffffff92b82000-0xffffffff92e58fff] [ 8.594427] Write protecting the kernel read-only data: 28672k [ 8.599361] debug: unmapping init [mem 0xffffffff91203000-0xffffffff913fffff] [ 8.603105] debug: unmapping init [mem 0xffffffff91b14000-0xffffffff91bfffff] [ 8.666621] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 8.676979] systemd[1]: Detected virtualization kvm. [ 8.680390] systemd[1]: Detected architecture x86-64. [ 8.682770] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 8.733594] systemd[1]: No hostname configured. [ 8.740877] systemd[1]: Set hostname to . [ 8.747500] random: systemd: uninitialized urandom read (16 bytes read) [ 8.754397] systemd[1]: Initializing machine ID from random generator. [ 8.919446] random: ln: uninitialized urandom read (6 bytes read) [ 9.162594] random: systemd: uninitialized urandom read (16 bytes read) [ 9.170481] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 9.197086] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 9.253450] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Timers. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 10.874527] device-mapper: uevent: version 1.0.3 [ 10.880471] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 12.412137] virtio_net virtio0 ens2: renamed from eth0 [ 12.571250] random: fast init done [ 12.832767] scsi host0: ata_piix [ 12.880352] scsi host1: ata_piix [ 12.884184] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 12.890588] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 18.084396] random: crng init done [ 18.086674] random: 7 urandom warning(s) missed due to ratelimiting [ 20.777300] dracut-initqueue[587]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 23.868785] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 27.947366] printk: systemd: 26 output lines suppressed due to ratelimiting [ 28.886729] SELinux: Disabled at runtime. [ 29.142634] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 29.178522] systemd[1]: Detected virtualization kvm. [ 29.190268] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 31.795073] systemd[1]: initrd-switch-root.service: Succeeded. [ 31.800627] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 31.821860] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 31.847807] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 31.853669] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 31.883434] systemd[1]: Starting Journal Service... Starting Journal Service... [ 31.901963] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ 32.809428] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 33.655592] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 34.733245] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 35.264727] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 35.439997] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 35.495470] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 40.296030] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 41.195824] NFS: Registering the id_resolver key type [ 41.203851] Key type id_resolver registered [ 41.209743] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg118-server login: [ 70.842264] hrtimer: interrupt took 2633380 ns [ 102.685289] libcfs: loading out-of-tree module taints kernel. [ 102.720545] Key type ._llcrypt registered [ 102.721851] Key type .llcrypt registered [ 102.778844] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 122.076918] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 124.068930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 124.096086] alg: No test for adler32 (adler32-zlib) [ 125.632962] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 126.721207] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 128.559556] Key type lgssc registered [ 131.460554] Lustre: Echo OBD driver; http://www.lustre.org/ [ 153.001088] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 196.228855] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 209.855406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 209.913931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 211.229525] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 211.274404] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 211.384038] Lustre: lustre-MDT0000: new disk, initializing [ 211.468378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 211.493783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 215.746766] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.575630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 228.686166] Lustre: 6517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 228.718602] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 228.723828] Lustre: Skipped 1 previous similar message [ 228.814563] Lustre: lustre-MDT0001: new disk, initializing [ 228.906883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 228.964537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 228.981373] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 234.877348] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 241.035057] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 250.177810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 250.385768] Lustre: lustre-OST0000: new disk, initializing [ 250.387952] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 250.392978] Lustre: 8456:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 250.446558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 257.065110] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 257.566913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 257.590906] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 257.655861] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 271.141343] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 271.350258] Lustre: lustre-OST0001: new disk, initializing [ 271.352893] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 271.370490] Lustre: 9530:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 271.450577] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 277.573559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 277.596207] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 277.701113] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 278.074136] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 290.410985] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 299.424837] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 306.223413] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing check_logdir /tmp/testlogs/ [ 314.066067] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing yml_node [ 319.517096] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 322.501735] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 325.444509] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 327.512294] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 13 09:23:50 EDT 2026 [ 347.186342] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 349.000599] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 367.072123] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 367.078765] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 367.107057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 370.146175] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 370.148076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 370.156995] Lustre: Skipped 2 previous similar messages [ 370.165876] Lustre: Skipped 2 previous similar messages [ 371.834711] Lustre: server umount lustre-MDT0000 complete [ 375.391173] LustreError: 10405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786627480 with bad export cookie 4310873599624622392 [ 375.392713] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 375.405265] LustreError: 10405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 380.384540] LustreError: 10356:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 380.385355] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 380.385685] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 380.406351] LustreError: 10356:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 380.417147] Lustre: Skipped 1 previous similar message [ 381.894500] Lustre: server umount lustre-MDT0001 complete [ 386.425608] Lustre: server umount lustre-OST0000 complete [ 391.585087] Lustre: server umount lustre-OST0001 complete [ 399.431845] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 410.908558] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 447.356565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 447.615774] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 447.652587] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 447.785597] Lustre: lustre-MDT0000: new disk, initializing [ 447.923665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 447.962674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 452.904656] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 463.719188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 463.901881] Lustre: 15085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 464.007716] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 464.013325] Lustre: Skipped 1 previous similar message [ 464.136289] Lustre: lustre-MDT0001: new disk, initializing [ 464.270117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 464.310424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 464.318552] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 470.048570] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 481.491594] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 488.580197] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 497.052597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 497.645842] Lustre: lustre-OST0000: new disk, initializing [ 497.651918] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 497.659341] Lustre: 17056:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 497.740756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 499.495486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 499.520217] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 499.693910] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 505.575548] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 516.036665] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 521.718304] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 521.963771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 527.080923] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 527.326331] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 530.408214] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 530.417040] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 530.438101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 530.457486] LustreError: Skipped 1 previous similar message [ 530.480852] Lustre: Skipped 2 previous similar messages [ 535.520965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 535.532381] Lustre: Skipped 1 previous similar message [ 535.804714] Lustre: server umount lustre-OST0000 complete [ 540.641609] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 540.656035] Lustre: Skipped 1 previous similar message [ 545.767108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 545.781550] Lustre: Skipped 3 previous similar messages [ 546.770071] Lustre: server umount lustre-MDT0000 complete [ 550.880945] LustreError: 15098:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.904960] LustreError: 15098:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 552.177788] LustreError: 15080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786627657 with bad export cookie 4310873599624623932 [ 552.191976] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 552.214500] LustreError: 15080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 552.841104] Lustre: server umount lustre-MDT0001 complete [ 560.612451] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 09:27:43 (1786627663) [ 582.712380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 583.353731] LustreError: 19932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 583.435622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 587.893881] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 588.770359] LustreError: 19933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 593.888889] LustreError: 19932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 598.378584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 604.010563] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 611.836564] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 616.904693] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 619.236439] Lustre: Failing over lustre-MDT0000 [ 619.503809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 619.517426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 619.531440] Lustre: Skipped 1 previous similar message [ 619.539207] LustreError: 20660:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 619.757522] Lustre: server umount lustre-MDT0000 complete [ 639.968511] Lustre: 3651:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786627728/real 1786627728] req@ffff90c987798700 x1873414435536512/t0(0) o400->MGC192.168.201.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786627744 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 639.970357] LustreError: 19932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 640.008616] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 640.053457] LustreError: 19932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 640.136397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 649.593783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 649.604412] Lustre: Skipped 1 previous similar message [ 649.683733] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 654.825841] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 654.841953] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 654.903890] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 655.329990] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 664.457989] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 670.115776] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 670.122574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 670.129855] Lustre: Skipped 2 previous similar messages [ 670.145275] Lustre: Skipped 1 previous similar message [ 675.297457] LustreError: 19928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 675.307770] LustreError: 19928:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 676.364764] Lustre: server umount lustre-MDT0000 complete [ 679.814571] LustreError: 19914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786627784 with bad export cookie 4310873599624625346 [ 679.824372] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 679.825503] LustreError: 19914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 680.187809] Lustre: server umount lustre-MDT0001 complete [ 688.182655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 693.768382] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 704.020674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 709.651983] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 715.901853] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 721.718773] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 723.859643] Lustre: Failing over lustre-MDT0000 [ 724.199909] Lustre: server umount lustre-MDT0000 complete [ 724.450802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 739.814190] LustreError: 23274:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 739.843104] LustreError: 23274:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 740.575119] Lustre: 3654:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786627829/real 1786627829] req@ffff90c987605180 x1873414435565696/t0(0) o400->MGC192.168.201.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786627845 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 740.587813] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 742.970707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 751.075033] LustreError: 3650:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff90c9875ac000 x1873414435568640/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 751.732591] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 756.708923] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 756.741562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 756.757267] Lustre: Skipped 1 previous similar message [ 756.801363] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 757.947631] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 769.888399] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 777.186267] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 777.193242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 780.711084] Lustre: server umount lustre-MDT0000 complete [ 784.439928] LustreError: 23255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786627889 with bad export cookie 4310873599624626284 [ 784.441228] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 784.456232] LustreError: 23255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 784.927868] Lustre: server umount lustre-MDT0001 complete [ 793.025554] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 09:31:36 (1786627896) [ 809.527484] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 819.327444] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 859.037479] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 871.892360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 872.129312] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 872.154498] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 872.208611] Lustre: lustre-MDT0000: new disk, initializing [ 872.274180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 872.280505] Lustre: Skipped 3 previous similar messages [ 872.291857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 875.723903] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 886.896431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 887.072558] Lustre: 29647:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 887.349984] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 892.041516] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 901.597770] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 906.979922] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 913.470263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 913.772189] Lustre: 31621:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 915.642387] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 915.686839] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 915.754392] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 915.762516] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 919.813533] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 929.785613] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 937.527865] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 956.899689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 956.915323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 956.936480] Lustre: Skipped 4 previous similar messages [ 956.964027] Lustre: Skipped 3 previous similar messages [ 957.541838] Lustre: server umount lustre-OST0000 complete [ 964.064871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 968.749963] Lustre: server umount lustre-MDT0000 complete [ 972.259612] LustreError: 29655:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 972.297197] LustreError: 29655:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 973.855961] LustreError: 29642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786628078 with bad export cookie 4310873599624626690 [ 973.860486] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 973.865446] LustreError: 29642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 974.603947] Lustre: server umount lustre-MDT0001 complete [ 994.808992] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1000.963773] Key type lgssc unregistered [ 1001.585411] LNet: 33625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1001.599532] LNetError: 33625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1002.661305] LNet: Removed LNI 192.168.201.118@tcp [ 1004.056703] Key type .llcrypt unregistered [ 1004.063269] Key type ._llcrypt unregistered [ 1036.552196] Key type ._llcrypt registered [ 1036.555471] Key type .llcrypt registered [ 1036.704380] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1038.228261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1038.332487] alg: No test for adler32 (adler32-zlib) [ 1039.529817] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 1039.932501] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 1041.639647] Key type lgssc registered [ 1042.898455] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1051.037394] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1051.057994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1052.474273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1057.473870] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1062.882926] LustreError: 35038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1062.903381] LustreError: 35038:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1066.683857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1067.112706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1071.995939] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1079.721321] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1085.151456] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1091.918714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1092.006242] LustreError: 36858:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1092.181697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1097.215057] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 1098.885649] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1107.253493] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1122.906139] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 09:37:05 (1786628225) [ 1124.982413] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1126.711407] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1133.628928] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 09:37:17 (1786628237) [ 1141.674698] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 09:37:24 (1786628244) [ 1142.903044] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1153.731122] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1153.740548] Lustre: Skipped 2 previous similar messages [ 1156.129088] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 09:37:38 (1786628258) [ 1158.617784] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1158.622340] Lustre: Skipped 2 previous similar messages [ 1167.559953] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1167.571340] Lustre: Skipped 4 previous similar messages [ 1184.607646] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 09:38:07 (1786628287) [ 1188.368344] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1193.396442] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1204.033122] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 09:38:26 (1786628306) [ 1206.296586] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1206.307649] Lustre: Skipped 6 previous similar messages [ 1219.637954] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 09:38:42 (1786628322) [ 1235.424731] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1235.436784] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1235.460618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1236.464364] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1236.466012] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1236.473148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1238.701211] Lustre: server umount lustre-OST0000 complete [ 1246.178775] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1246.184277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1246.197747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1246.205923] Lustre: Skipped 1 previous similar message [ 1248.369890] Lustre: server umount lustre-MDT0000 complete [ 1251.811903] LustreError: 37528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1251.831953] LustreError: 37528:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1252.604838] LustreError: 38955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786628357 with bad export cookie 9453536188275597408 [ 1252.615262] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1252.932195] Lustre: server umount lustre-MDT0001 complete [ 1264.813953] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1267.887214] Key type lgssc unregistered [ 1268.272711] LNet: 42896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1268.280957] LNetError: 42896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1268.294947] LNet: Removed LNI 192.168.201.118@tcp [ 1269.108916] Key type .llcrypt unregistered [ 1269.110875] Key type ._llcrypt unregistered [ 1288.230726] Key type ._llcrypt registered [ 1288.233604] Key type .llcrypt registered [ 1288.383269] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 1304.440854] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1305.790945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1305.864284] alg: No test for adler32 (adler32-zlib) [ 1306.937934] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 1307.208841] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 1308.879270] Key type lgssc registered [ 1310.290829] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1348.137258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1348.191291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1349.575849] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1349.607400] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1349.647307] Lustre: lustre-MDT0000: new disk, initializing [ 1349.680514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1349.694528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1356.617537] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1370.390959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1370.596245] Lustre: 46054:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1370.729713] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1370.733776] Lustre: Skipped 1 previous similar message [ 1370.856770] Lustre: lustre-MDT0001: new disk, initializing [ 1371.068682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1371.158938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1371.177907] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1375.420314] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1387.489989] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1393.773771] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1400.882772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1401.265388] Lustre: lustre-OST0000: new disk, initializing [ 1401.277303] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1401.293623] Lustre: 48019:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1401.426380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1408.309570] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1408.615073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1408.629911] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1408.736497] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1418.752458] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1424.534727] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1424.744725] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1430.217267] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1430.512544] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1434.593462] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1434.593567] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1434.607505] LustreError: Skipped 1 previous similar message [ 1434.618556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1434.628858] Lustre: Skipped 1 previous similar message [ 1434.658512] Lustre: Skipped 1 previous similar message [ 1438.995382] Lustre: server umount lustre-OST0000 complete [ 1444.832551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1444.848485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1444.870393] Lustre: Skipped 1 previous similar message [ 1449.952536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1449.966739] Lustre: Skipped 2 previous similar messages [ 1451.003164] Lustre: server umount lustre-MDT0000 complete [ 1455.460610] LustreError: 49762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786628560 with bad export cookie 1080260686997287692 [ 1455.464726] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1455.475540] LustreError: 49762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1455.813029] Lustre: server umount lustre-MDT0001 complete [ 1467.237925] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 09:42:49 (1786628569) [ 1474.477673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1474.860043] LustreError: 50480:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1474.876373] LustreError: 50480:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1474.951716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1480.161743] LustreError: 50481:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1480.551889] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1484.259609] LustreError: 50480:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1489.336499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1489.379531] LustreError: 50481:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1489.847075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1494.083513] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1502.182434] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1508.302673] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1515.601705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1516.072250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1523.531813] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1530.923579] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1538.719887] LustreError: 53008:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1539.618398] Lustre: Modifying parameter general.timeout=1 in log params [ 1541.655652] Lustre: Modifying parameter general.timeout=3 in log params [ 1541.662675] Lustre: Skipped 1 previous similar message [ 1544.452553] Lustre: Modifying parameter general.timeout=6 in log params [ 1544.458981] Lustre: Skipped 2 previous similar messages [ 1549.444165] Lustre: Modifying parameter general.timeout=11 in log params [ 1549.449906] Lustre: Skipped 4 previous similar messages [ 1557.741843] Lustre: Modifying parameter general.timeout=19 in log params [ 1557.753361] Lustre: Skipped 7 previous similar messages [ 1574.584934] Lustre: Modifying parameter general.timeout=38 in log params [ 1574.592883] Lustre: Skipped 18 previous similar messages [ 1606.671487] Lustre: Modifying parameter general.timeout=69 in log params [ 1606.675421] Lustre: Skipped 30 previous similar messages [ 1671.176863] Lustre: Disabling parameter general.timeout= in log params [ 1671.181161] Lustre: Skipped 60 previous similar messages [ 1674.565400] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 09:46:18 (1786628778) [ 1677.314474] Lustre: 59953:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1679.272476] Lustre: 60053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1683.085315] Lustre: 60249:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1683.096238] Lustre: 60249:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1713.763757] Lustre: 61824:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1713.772042] Lustre: 61824:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1713.794332] LustreError: 61824:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1718.019502] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 09:47:01 (1786628821) [ 1735.649828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1735.655091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1735.685616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1740.031709] Lustre: server umount lustre-MDT0000 complete [ 1741.798374] LustreError: 50476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1741.804830] LustreError: 50476:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1743.969331] LustreError: 50462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786628848 with bad export cookie 1080260686997288574 [ 1743.976175] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1743.994954] LustreError: 50462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1744.399660] Lustre: server umount lustre-MDT0001 complete [ 1759.125091] Lustre: server umount lustre-OST0000 complete [ 1786.903134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1787.283802] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1787.611895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1791.847481] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1799.976133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1800.183622] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1800.260234] Lustre: 64356:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1800.496993] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1800.499773] Lustre: Skipped 3 previous similar messages [ 1805.249151] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1812.995850] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1818.622094] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1826.408572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1826.669097] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1826.916702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1826.927763] Lustre: Skipped 1 previous similar message [ 1833.111521] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1841.901812] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1844.017911] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1844.036631] Lustre: Skipped 23 previous similar messages [ 1844.128371] Lustre: 66919:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1851.360712] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1851.368440] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1851.377417] Lustre: Skipped 2 previous similar messages [ 1851.390878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1857.502090] Lustre: server umount lustre-OST0000 complete [ 1862.124108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1862.157631] Lustre: Skipped 2 previous similar messages [ 1867.575497] Lustre: server umount lustre-MDT0000 complete [ 1872.081915] LustreError: 64347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786628977 with bad export cookie 1080260686997291605 [ 1872.095734] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1872.501724] Lustre: server umount lustre-MDT0001 complete [ 1884.850046] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1887.892366] Key type lgssc unregistered [ 1888.202633] LNet: 68129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1888.212820] LNetError: 68129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1888.247524] LNet: Removed LNI 192.168.201.118@tcp [ 1889.242668] Key type .llcrypt unregistered [ 1889.248122] Key type ._llcrypt unregistered [ 1915.094463] Key type ._llcrypt registered [ 1915.098779] Key type .llcrypt registered [ 1915.226225] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1916.694123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1916.734462] alg: No test for adler32 (adler32-zlib) [ 1917.746980] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 1917.990438] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 1919.719163] Key type lgssc registered [ 1920.698652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1928.070736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1928.107431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1929.542302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1934.955657] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1939.942674] LustreError: 69546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1939.967173] LustreError: 69546:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1944.071399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1945.075966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1950.499159] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1960.137035] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1966.319168] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1973.814448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1974.543979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1981.747394] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1989.518032] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1997.615660] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2005.984622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2005.993107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2006.017690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2010.596501] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2010.603741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2010.627476] Lustre: Skipped 1 previous similar message [ 2012.063083] Lustre: server umount lustre-MDT0000 complete [ 2015.735406] LustreError: 69543:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2015.761178] LustreError: 69543:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2016.522083] LustreError: 69526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786629121 with bad export cookie 7082595831263661431 [ 2016.559769] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2016.848800] Lustre: server umount lustre-MDT0001 complete [ 2031.453273] Lustre: server umount lustre-OST0000 complete [ 2064.272535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2064.720440] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2064.780484] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2065.147761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2070.105227] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2079.180906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2079.319518] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2079.363045] Lustre: 74418:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2079.826992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2084.608541] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2091.853976] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2097.822758] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2104.026501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2104.140116] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2104.422146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2110.671242] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2117.596815] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2119.442279] Lustre: 76991:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2121.129553] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2129.377786] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2129.387861] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2129.403781] Lustre: Skipped 1 previous similar message [ 2129.417958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2129.428447] Lustre: Skipped 1 previous similar message [ 2132.247931] Lustre: server umount lustre-OST0000 complete [ 2136.042339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2136.056607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2136.074127] Lustre: Skipped 1 previous similar message [ 2136.087693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2144.739868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2144.748970] Lustre: Skipped 3 previous similar messages [ 2150.367635] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2150.598214] Lustre: server umount lustre-MDT0000 complete [ 2154.048259] LustreError: 76993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786629258 with bad export cookie 7082595831263662271 [ 2154.060140] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2154.061683] LustreError: 76993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2154.448896] Lustre: server umount lustre-MDT0001 complete [ 2165.811396] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 2168.578864] Key type lgssc unregistered [ 2168.831665] LNet: 78210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2168.837465] LNetError: 78210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2168.859281] LNet: Removed LNI 192.168.201.118@tcp [ 2169.725869] Key type .llcrypt unregistered [ 2169.727699] Key type ._llcrypt unregistered [ 2196.391982] Key type ._llcrypt registered [ 2196.396352] Key type .llcrypt registered [ 2196.531886] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2197.946493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2198.025621] alg: No test for adler32 (adler32-zlib) [ 2199.208845] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 2199.645681] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 2201.447207] Key type lgssc registered [ 2203.062738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2211.012149] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2211.033596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2212.635211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2217.080782] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2223.080415] LustreError: 79626:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2223.095313] LustreError: 79626:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2225.200482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2225.847416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2229.947406] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2236.913165] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2242.909527] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2249.358555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2249.999957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2257.046845] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2265.654525] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2272.516217] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2273.386068] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2275.331299] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2275.335293] Lustre: Skipped 1 previous similar message [ 2277.983620] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2277.989993] Lustre: Skipped 2 previous similar messages [ 2282.852572] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2282.859739] Lustre: Skipped 4 previous similar messages [ 2291.280538] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2291.287747] Lustre: Skipped 8 previous similar messages [ 2307.397498] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2307.401173] Lustre: Skipped 16 previous similar messages [ 2339.811203] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2339.818437] Lustre: Skipped 35 previous similar messages [ 2372.772818] Lustre: 87730:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2376.282783] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 09:57:59 (1786629479) [ 2378.050155] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2380.644540] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 09:58:03 (1786629483) [ 2386.400589] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2386.405914] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2386.410354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2386.439285] Lustre: Skipped 1 previous similar message [ 2391.554092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2391.567383] Lustre: Skipped 2 previous similar messages [ 2392.564893] Lustre: server umount lustre-OST0000 complete [ 2399.722602] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2399.726352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2399.733954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2401.763101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2401.764549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2401.785381] Lustre: Skipped 1 previous similar message [ 2403.763189] Lustre: server umount lustre-MDT0000 complete [ 2406.882112] LustreError: 80357:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2406.902380] LustreError: 80357:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2407.473561] LustreError: 79607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786629512 with bad export cookie 1325824132851070701 [ 2407.475274] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2407.484445] LustreError: 79607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2408.006826] Lustre: server umount lustre-MDT0001 complete [ 2419.145773] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 2421.529374] Key type lgssc unregistered [ 2421.753964] LNet: 88981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2421.770432] LNetError: 88981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2421.800234] LNet: Removed LNI 192.168.201.118@tcp [ 2422.558803] Key type .llcrypt unregistered [ 2422.563888] Key type ._llcrypt unregistered [ 2427.763659] Key type ._llcrypt registered [ 2427.765996] Key type .llcrypt registered [ 2427.874248] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2434.458926] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules [ 2436.185488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2436.588931] alg: No test for adler32 (adler32-zlib) [ 2437.880588] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 2438.383876] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 2440.056314] Key type lgssc registered [ 2442.168876] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2444.578636] LustreError: 90255:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2462.525213] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2465.836228] LustreError: 90255:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2474.606028] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2474.649215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2476.371848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2481.469735] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2486.752891] LustreError: 91262:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2486.790067] LustreError: 91262:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2491.479577] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 09:59:54 (1786629594) [ 2491.872641] LustreError: 91261:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2496.993073] LustreError: 91262:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2497.708685] LustreError: 91293:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2497.728812] LustreError: 91293:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2497.734910] LustreError: 91293:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2498.456773] Lustre: server umount lustre-MDT0000 complete [ 2515.038064] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 2518.924782] Key type lgssc unregistered [ 2519.357179] LNet: 92791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2519.367677] LNetError: 92791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2519.404364] LNet: Removed LNI 192.168.201.118@tcp [ 2520.894482] Key type .llcrypt unregistered [ 2520.903360] Key type ._llcrypt unregistered [ 2542.494530] Key type ._llcrypt registered [ 2542.496409] Key type .llcrypt registered [ 2542.570820] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2543.433743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2543.463184] alg: No test for adler32 (adler32-zlib) [ 2544.606301] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 2544.878653] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 2546.527250] Key type lgssc registered [ 2547.661950] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2555.371669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2555.412475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2557.132616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2561.407121] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2566.630355] LustreError: 94022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2566.663941] LustreError: 94022:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2569.368074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2570.545993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2575.049241] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2582.431347] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2587.771923] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2594.715819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2595.317367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2602.031749] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2610.568151] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2659.348986] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 10:02:42 (1786629762) [ 2661.079969] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2663.032388] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 10:02:46 (1786629766) [ 2671.074743] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2671.077243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2671.120333] Lustre: Skipped 2 previous similar messages [ 2671.138965] Lustre: Skipped 1 previous similar message [ 2674.437444] Lustre: server umount lustre-MDT0000 complete [ 2676.196080] LustreError: 94017:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2676.206837] LustreError: 94017:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2679.086028] LustreError: 94764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786629784 with bad export cookie 5767834018103308515 [ 2679.092467] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2679.110445] LustreError: 94764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2679.761711] Lustre: server umount lustre-MDT0001 complete [ 2695.475826] Lustre: server umount lustre-OST0000 complete [ 2706.725626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2707.480361] LustreError: 98296:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2707.654352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2712.897958] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2713.066913] LustreError: 98297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2717.155914] LustreError: 98296:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2721.654229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2722.324339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2727.096747] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2734.167099] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2740.285383] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2755.465796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2755.663311] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2755.669934] LustreError: 98283:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2755.691957] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2755.700155] LustreError: 100379:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2755.703506] LustreError: 100379:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2755.708701] LustreError: 100379:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2755.772405] Lustre: server umount lustre-OST0000 complete [ 2755.775900] LustreError: 100379:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2766.081046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2766.255959] LustreError: 100848:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2766.364402] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2766.374970] LustreError: 98283:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2766.404526] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2766.416095] LustreError: 100844:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2766.427172] LustreError: 100844:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2766.430201] LustreError: 100844:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2766.529543] Lustre: server umount lustre-OST0000 complete [ 2766.533323] LustreError: 100844:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2776.911926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2777.067270] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2777.070651] Lustre: Found index 0 for lustre-OST0000, updating log [ 2777.075810] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2777.181638] Lustre: lustre-OST0000: new disk, initializing [ 2777.184823] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2777.191642] Lustre: 101380:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2777.388528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2783.071228] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2786.272650] Lustre: Failing over lustre-OST0000 [ 2786.432476] Lustre: server umount lustre-OST0000 complete [ 2793.953707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2793.957454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2793.964472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2793.964482] Lustre: Skipped 1 previous similar message [ 2797.027019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2797.038350] Lustre: Skipped 2 previous similar messages [ 2800.213410] Lustre: server umount lustre-MDT0000 complete [ 2802.145263] LustreError: 98296:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2802.167586] LustreError: 98296:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2804.055620] LustreError: 98276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786629908 with bad export cookie 5767834018103311259 [ 2804.066796] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2804.072590] LustreError: 98276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2804.417700] Lustre: server umount lustre-MDT0001 complete [ 2814.930716] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 10:05:18 (1786629918) [ 2832.580643] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2844.506711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2844.889729] LustreError: 103728:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2845.027210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2850.343486] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2859.821300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2866.408941] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2869.650117] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2876.460369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2877.034083] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2883.190850] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2885.231780] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2885.245164] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2885.269152] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2889.395742] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:06 [ 2889.418868] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2889.429588] Lustre: lustre-OST0000: Recovery over after 0:04, of 1 clients 1 recovered and 0 were evicted. [ 2889.434326] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2889.445460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2891.079067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2891.178668] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2891.236198] Lustre: lustre-OST0001: new disk, initializing [ 2891.238969] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2891.243061] Lustre: 106051:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2891.539902] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2891.544190] Lustre: Skipped 2 previous similar messages [ 2897.531305] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2899.092962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2899.109782] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2899.256160] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2905.084568] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2913.488279] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2919.911562] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2919.913429] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2919.934258] Lustre: Skipped 3 previous similar messages [ 2919.960961] Lustre: Skipped 2 previous similar messages [ 2924.059738] Lustre: server umount lustre-MDT0001 complete [ 2925.030762] LustreError: 103725:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2925.051978] LustreError: 103725:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 2934.999294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2935.112189] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2935.124289] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2935.211566] Lustre: 103717:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2936.889368] LustreError: 107621:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2936.896860] LustreError: 107621:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2936.903166] LustreError: 103720:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2937.027287] LustreError: 107633:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.118@tcp: cfg command failed: rc = -17 [ 2937.045062] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2937.045062] [ 2940.692318] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2940.922454] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2950.153331] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 10:07:33 (1786630053) [ 2967.811326] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2985.229446] Lustre: 109674:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3011.318355] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3015.608618] Lustre: 110809:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3099.615593] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3099.618128] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3099.618954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3099.618960] Lustre: Skipped 1 previous similar message [ 3099.654489] Lustre: Skipped 2 previous similar messages [ 3101.556082] Lustre: server umount lustre-MDT0000 complete [ 3104.741046] LustreError: 103723:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3104.766816] LustreError: 103723:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 3106.447063] LustreError: 106770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786630211 with bad export cookie 5767834018103312260 [ 3106.448884] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3106.463599] LustreError: 106770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3106.832616] Lustre: server umount lustre-MDT0001 complete [ 3120.975740] Lustre: server umount lustre-OST0000 complete [ 3125.215155] Lustre: 93322:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786630214/real 1786630214] req@ffff90c98f330000 x1873416971185024/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786630230 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3125.245472] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3125.263917] Lustre: Skipped 1 previous similar message [ 3125.652569] Lustre: server umount lustre-OST0001 complete [ 3132.276645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3144.000488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3153.577991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3165.085267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3188.697574] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3200.889261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3200.912933] Lustre: lustre-MDT0000: reset Object Index mappings [ 3201.483677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3201.490723] Lustre: Skipped 1 previous similar message [ 3206.198936] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3214.471479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3214.495189] Lustre: lustre-MDT0001: reset Object Index mappings [ 3220.593927] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3223.848623] Lustre: 115905:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3231.077049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3239.861036] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 3239.952755] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3247.109336] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 3250.616375] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3256.833553] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3256.834053] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 3258.501946] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3267.386914] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3270.915765] Lustre: 117783:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3318.036958] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 10:13:41 (1786630421) [ 3347.938513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3347.949195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3347.974217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3347.985272] Lustre: Skipped 3 previous similar messages [ 3353.134852] Lustre: server umount lustre-MDT0000 complete [ 3357.152061] LustreError: 114738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786630462 with bad export cookie 5767834018103359356 [ 3357.156420] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3357.176980] LustreError: 114738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3362.209217] LustreError: 114753:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3362.248614] LustreError: 114753:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 3364.047586] Lustre: server umount lustre-MDT0001 complete [ 3383.263178] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3383.361285] Lustre: server umount lustre-OST0000 complete [ 3387.904171] LustreError: 118741:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0001_UUID: namespace resource [0x2c0000400:0xb5:0x0].0x0 (ffff90c9844d5800) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3402.207193] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3402.577053] Lustre: server umount lustre-OST0001 complete [ 3412.357792] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 3424.361525] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3473.201415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3473.560423] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3473.624786] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3473.727583] Lustre: lustre-MDT0000: new disk, initializing [ 3473.854734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3473.857753] Lustre: Skipped 3 previous similar messages [ 3473.902560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3481.212558] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3490.436912] Lustre: server umount lustre-MDT0000 complete [ 3502.487938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3503.048692] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3503.087453] Lustre: 122057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3503.125654] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3509.026809] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3514.082162] Lustre: server umount lustre-MDT0000 complete [ 3522.751585] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 10:17:05 (1786630625) [ 3525.013784] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3526.772558] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 10:17:10 (1786630630) [ 3546.298311] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3557.188219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3563.157773] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3572.213982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3572.421681] Lustre: 124108:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3572.475608] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3572.480046] Lustre: Skipped 1 previous similar message [ 3572.598566] Lustre: lustre-MDT0001: new disk, initializing [ 3572.773419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3572.781581] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3577.397777] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3582.540542] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3590.390559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3590.586885] Lustre: lustre-OST0000: new disk, initializing [ 3590.589343] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3590.593381] Lustre: 125683:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3591.774613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3591.794683] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3591.959418] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3597.905675] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3613.126376] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3613.537243] Lustre: lustre-OST0001: new disk, initializing [ 3613.540172] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3613.552712] Lustre: 126556:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3615.284953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3615.344751] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3615.608715] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3622.332664] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3632.490431] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3636.080687] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4005.136878] Lustre: 132967:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4005.142474] Lustre: 132967:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4005.148684] Lustre: 132967:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4005.155845] Lustre: 132967:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4005.160213] Lustre: 132967:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4005.164815] Lustre: 132967:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4297.690781] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:30:00 (1786631400) [ 4315.107896] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4315.110435] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4315.111205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4315.111211] Lustre: Skipped 8 previous similar messages [ 4315.125907] Lustre: Skipped 8 previous similar messages [ 4319.423559] Lustre: server umount lustre-MDT0000 complete [ 4320.250239] LustreError: 124113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4320.289665] LustreError: 124113:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4323.752366] LustreError: 124100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786631428 with bad export cookie 5767834018103425716 [ 4323.762835] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4323.778329] LustreError: 124100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4325.353471] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4325.372212] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4325.398308] Lustre: Skipped 1 previous similar message [ 4325.426060] Lustre: Skipped 4 previous similar messages [ 4333.496422] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4333.506601] Lustre: Skipped 3 previous similar messages [ 4338.143469] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4338.644938] Lustre: server umount lustre-MDT0001 complete [ 4357.088992] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4357.266168] Lustre: server umount lustre-OST0000 complete [ 4367.526981] Lustre: server umount lustre-OST0001 complete [ 4374.847749] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 4385.656631] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 4426.024839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4426.398250] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4426.464166] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4426.581106] Lustre: lustre-MDT0000: new disk, initializing [ 4426.698364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4426.709234] Lustre: Skipped 5 previous similar messages [ 4426.743228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4432.688377] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4444.708198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4444.845604] Lustre: 136024:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4444.882983] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4444.886302] Lustre: Skipped 1 previous similar message [ 4444.949508] Lustre: lustre-MDT0001: new disk, initializing [ 4445.123444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4445.141165] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4450.425438] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4461.752423] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4467.115672] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4474.268430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4474.642158] Lustre: lustre-OST0000: new disk, initializing [ 4474.645728] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4474.654241] Lustre: 137993:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4474.781440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4474.786704] Lustre: Skipped 1 previous similar message [ 4475.840686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4475.852773] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4476.054884] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4481.764828] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4491.158966] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4500.872400] Lustre: lustre-MDD0000: changelog on [ 4503.585786] Lustre: lustre-MDD0001: changelog on [ 6205.229726] Lustre: lustre-MDD0001: changelog off [ 6206.790095] Lustre: 136032:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6207.629050] Lustre: lustre-MDD0000: changelog off [ 6216.672963] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6216.676916] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6216.691372] LustreError: Skipped 1 previous similar message [ 6216.695526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6216.731445] Lustre: Skipped 1 previous similar message [ 6221.801841] LustreError: 139624:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6221.833710] LustreError: 139624:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 6221.925626] Lustre: server umount lustre-OST0000 complete [ 6226.912873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6226.915137] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6226.923401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6226.932697] Lustre: Skipped 1 previous similar message [ 6226.943661] Lustre: Skipped 2 previous similar messages [ 6232.036859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6232.052565] Lustre: Skipped 1 previous similar message [ 6232.551282] Lustre: server umount lustre-MDT0000 complete [ 6236.143988] LustreError: 136015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786633341 with bad export cookie 5767834018104246270 [ 6236.145617] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6236.158477] LustreError: 136015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6236.584666] Lustre: server umount lustre-MDT0001 complete [ 6247.914201] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 6250.766528] Key type lgssc unregistered [ 6251.018510] LNet: 147109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6251.028882] LNetError: 147109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6251.058126] LNet: Removed LNI 192.168.201.118@tcp [ 6252.002137] Key type .llcrypt unregistered [ 6252.003917] Key type ._llcrypt unregistered [ 6271.825270] Key type ._llcrypt registered [ 6271.826619] Key type .llcrypt registered [ 6271.935552] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 6288.099469] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 6289.331410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6289.367647] alg: No test for adler32 (adler32-zlib) [ 6290.474220] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 6290.769202] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 6292.527197] Key type lgssc registered [ 6293.848238] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6333.994459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6334.026242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6335.423271] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6335.478740] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6335.615539] Lustre: lustre-MDT0000: new disk, initializing [ 6335.765722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6335.822878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6341.774839] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6352.933292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6353.160385] Lustre: 150313:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6353.212235] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6353.214554] Lustre: Skipped 1 previous similar message [ 6353.275328] Lustre: lustre-MDT0001: new disk, initializing [ 6353.345573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6353.378900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6353.389781] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6357.871209] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6367.460550] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6373.801480] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6380.586891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6380.901952] Lustre: lustre-OST0000: new disk, initializing [ 6380.906394] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6380.919934] Lustre: 152283:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6381.037287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6386.662051] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6388.829674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6388.852566] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6388.965786] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6395.839503] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6401.192362] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_module obdecho/obdecho [ 6402.990683] LustreError: 153287:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6403.005928] LustreError: 153287:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6404.692416] LustreError: 153383:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6414.818557] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6414.821663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6414.828444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6414.845677] Lustre: Skipped 1 previous similar message [ 6419.938581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6419.952203] Lustre: Skipped 4 previous similar messages [ 6421.260756] Lustre: server umount lustre-MDT0000 complete [ 6425.291620] LustreError: 150305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786633530 with bad export cookie 4073096276759238024 [ 6425.299661] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6425.306712] LustreError: 150305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6425.789289] Lustre: server umount lustre-MDT0001 complete [ 6440.978868] Lustre: server umount lustre-OST0000 complete [ 6463.584309] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 6476.078249] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 6515.863573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6516.261254] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6516.295368] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6516.382233] Lustre: lustre-MDT0000: new disk, initializing [ 6516.577913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6516.625124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6521.778180] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6534.870825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6535.120302] Lustre: 157100:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6535.291455] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6535.296679] Lustre: Skipped 1 previous similar message [ 6535.447552] Lustre: lustre-MDT0001: new disk, initializing [ 6535.580923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6535.646834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6535.664650] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6540.760819] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6553.642586] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6560.954131] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6568.281855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6568.634794] Lustre: lustre-OST0000: new disk, initializing [ 6568.642941] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6568.662246] Lustre: 159071:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6568.790941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6574.651463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6574.667626] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6574.817890] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6575.365879] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6584.813801] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6589.998161] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6590.195986] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6595.302110] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6595.576975] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6600.690524] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6600.691637] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6600.697192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6600.730418] Lustre: Skipped 2 previous similar messages [ 6604.413179] Lustre: server umount lustre-OST0000 complete [ 6610.912781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6610.917102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6610.940364] Lustre: Skipped 1 previous similar message [ 6610.959397] Lustre: Skipped 2 previous similar messages [ 6616.032862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6623.199151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6623.490980] Lustre: server umount lustre-MDT0000 complete [ 6625.252676] LustreError: 157113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6625.280251] LustreError: 157113:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6627.634970] LustreError: 157092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786633732 with bad export cookie 4073096276759238983 [ 6627.644907] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6628.444762] Lustre: server umount lustre-MDT0001 complete [ 6632.940321] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 11:08:55 (1786633735) [ 6634.963270] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6636.995959] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 11:09:00 (1786633740) [ 6644.790043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6645.262573] LustreError: 161486:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6645.351411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6650.682671] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6650.848878] LustreError: 161487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6655.973143] LustreError: 161486:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6660.464634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6666.198648] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6674.180601] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6681.401390] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6689.113662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6689.447306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6689.456470] Lustre: Skipped 1 previous similar message [ 6696.450167] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6705.675273] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6711.530284] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.18@tcp (stopping) [ 6711.536786] Lustre: Skipped 3 previous similar messages [ 6712.289337] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6712.294497] LustreError: Skipped 1 previous similar message [ 6712.300363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6716.385368] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6716.414177] Lustre: Skipped 1 previous similar message [ 6717.360981] Lustre: server umount lustre-MDT0000 complete [ 6721.508240] LustreError: 161487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6721.528695] LustreError: 161487:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6722.381634] LustreError: 161468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786633827 with bad export cookie 4073096276759239865 [ 6722.386186] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6722.394856] LustreError: 161468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6726.631856] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6727.671570] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6727.682916] Lustre: Skipped 7 previous similar messages [ 6729.012682] Lustre: server umount lustre-MDT0001 complete [ 6737.437333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6751.365663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6751.591248] LustreError: 165331:0:(llog.c:1655:llog_backup()) MGC192.168.201.118@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6751.602800] Lustre: 165331:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.118@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6752.225221] LustreError: 165336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6752.246422] LustreError: 165336:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6772.704594] LustreError: 165338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6772.725860] LustreError: 165338:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 6807.695438] LustreError: 165337:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6807.710407] LustreError: 165337:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 25 previous similar messages [ 6812.639753] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6812.645708] Lustre: 165331:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.118@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6853.647604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6858.729279] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6858.980304] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6868.417156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6873.060428] LustreError: 165337:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6873.084497] LustreError: 165337:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 44 previous similar messages [ 6888.421367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6892.578911] LustreError: 148534:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006f8a7627 x1873420898931072/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6892.684384] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.18@tcp (not set up) [ 6897.008774] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6898.187356] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6904.335465] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6905.558521] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6911.592118] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6913.525802] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6930.037544] Lustre: server umount lustre-MDT0000 complete [ 6933.959990] LustreError: 161467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786634038 with bad export cookie 4073096276759246354 [ 6933.969497] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6933.996340] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6934.340792] Lustre: server umount lustre-MDT0001 complete [ 6938.449416] Lustre: server umount lustre-OST0000 complete [ 6961.716743] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 6974.053709] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 7011.899813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7012.283277] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7012.330850] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7012.492751] Lustre: lustre-MDT0000: new disk, initializing [ 7012.590610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7012.593616] Lustre: Skipped 1 previous similar message [ 7012.601373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7017.570809] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7029.259245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7029.403464] Lustre: 170438:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7029.661956] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7034.166198] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7044.449459] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7049.906220] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7057.200150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7057.682288] Lustre: lustre-OST0000: new disk, initializing [ 7057.684687] Lustre: Skipped 1 previous similar message [ 7057.690479] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7057.694885] Lustre: Skipped 2 previous similar messages [ 7057.699848] Lustre: 172406:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7064.284164] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7066.227851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7066.244338] Lustre: Skipped 1 previous similar message [ 7066.275530] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7066.324524] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7074.996679] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7080.634766] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7080.889576] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7086.375571] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7086.548352] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7092.192966] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7092.196510] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7092.202528] Lustre: Skipped 1 previous similar message [ 7092.207503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7092.212331] Lustre: Skipped 2 previous similar messages [ 7095.055287] Lustre: server umount lustre-OST0000 complete [ 7101.424178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7101.432265] LustreError: Skipped 1 previous similar message [ 7105.743607] Lustre: server umount lustre-MDT0000 complete [ 7107.559368] LustreError: 170450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7107.582548] LustreError: 170450:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [ 7109.758564] LustreError: 170429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786634214 with bad export cookie 4073096276759263154 [ 7109.761419] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7109.772691] LustreError: 170429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7110.212059] Lustre: server umount lustre-MDT0001 complete [ 7119.085539] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 11:17:02 (1786634222) [ 7125.444714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7130.424526] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7139.007539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7143.498586] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7150.659453] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7156.615640] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7163.400289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7169.563536] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7177.086695] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7186.630278] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 11:18:10 (1786634290) [ 7195.621578] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7195.634510] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7195.653728] Lustre: Skipped 3 previous similar messages [ 7198.136277] Lustre: server umount lustre-OST0000 complete [ 7208.393352] Lustre: server umount lustre-MDT0000 complete [ 7212.739061] LustreError: 174798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786634317 with bad export cookie 4073096276759264036 [ 7212.740096] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7212.747019] LustreError: 174798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7213.407201] Lustre: server umount lustre-MDT0001 complete [ 7226.402541] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 7229.883162] Key type lgssc unregistered [ 7230.279972] LNet: 178549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7230.293451] LNetError: 178549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7230.308977] LNet: Removed LNI 192.168.201.118@tcp [ 7231.343456] Key type .llcrypt unregistered [ 7231.345649] Key type ._llcrypt unregistered [ 7254.493420] Key type ._llcrypt registered [ 7254.495665] Key type .llcrypt registered [ 7254.649688] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 7256.294465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7256.351376] alg: No test for adler32 (adler32-zlib) [ 7257.638714] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 7257.978984] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 7259.655170] Key type lgssc registered [ 7261.299785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7269.685911] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7269.702988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7285.216407] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7290.335415] LustreError: 179793:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.118@tcp: failed processing log, type 4: rc = -110 [ 7295.456148] LustreError: 179793:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7295.466945] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7295.473916] LustreError: MGC192.168.201.118@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7295.482400] LustreError: 179793:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7295.487270] LustreError: 179793:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7295.493489] LustreError: 179793:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7295.572948] Lustre: server umount lustre-OST0000 complete [ 7295.578740] LustreError: 179793:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7305.315988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7306.686739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7311.282228] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7316.962962] LustreError: 180293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7316.978356] LustreError: 180293:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7322.082339] LustreError: 180292:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7322.159565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7322.671274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7328.028929] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7335.476296] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7340.638626] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7348.435993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7348.594545] LustreError: 182111:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7348.604321] Lustre: 182111:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.118@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7348.733514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7355.152474] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7364.602522] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7368.672376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7368.676270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7368.684337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7372.257976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7372.272635] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7372.287542] Lustre: Skipped 1 previous similar message [ 7374.389626] Lustre: server umount lustre-MDT0000 complete [ 7377.383500] LustreError: 180293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7377.400840] LustreError: 180293:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7378.551115] LustreError: 180274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786634483 with bad export cookie 2822878346904931075 [ 7378.559739] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7378.571482] LustreError: 180274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7379.160149] Lustre: server umount lustre-MDT0001 complete [ 7394.220648] Lustre: server umount lustre-OST0000 complete [ 7408.024971] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 7422.703580] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 7460.984497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7461.257143] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7461.300803] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7461.403077] Lustre: lustre-MDT0000: new disk, initializing [ 7461.497288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7461.520934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7466.423339] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7477.011574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7477.153323] Lustre: 185656:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7477.256536] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7477.263562] Lustre: Skipped 1 previous similar message [ 7477.390603] Lustre: lustre-MDT0001: new disk, initializing [ 7477.517722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7477.582179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7477.596234] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7482.592924] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7492.858487] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7498.775790] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7507.011663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7507.519885] Lustre: lustre-OST0000: new disk, initializing [ 7507.530766] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7507.538214] Lustre: 187623:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7507.718982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7514.306521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7514.330782] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7514.543577] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7516.560810] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7528.255495] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7534.226357] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7534.378544] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7539.734890] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7540.031602] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7545.316058] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7545.317012] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7545.328753] LustreError: Skipped 1 previous similar message [ 7545.340301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7545.355782] Lustre: Skipped 1 previous similar message [ 7545.356879] Lustre: Skipped 2 previous similar messages [ 7548.171285] Lustre: server umount lustre-OST0000 complete [ 7554.528549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7554.540368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7554.556622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7554.566118] Lustre: Skipped 1 previous similar message [ 7558.718110] Lustre: server umount lustre-MDT0000 complete [ 7560.678575] LustreError: 185662:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7560.689308] LustreError: 185662:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7562.592151] LustreError: 185649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786634667 with bad export cookie 2822878346904931796 [ 7562.594031] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7562.603827] LustreError: 185649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7563.126646] Lustre: server umount lustre-MDT0001 complete [ 7572.800399] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 11:24:36 (1786634676) [ 7590.965834] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 7593.752248] Key type lgssc unregistered [ 7594.136403] LNet: 190573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7594.142562] LNetError: 190573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7594.189240] LNet: Removed LNI 192.168.201.118@tcp [ 7595.201586] Key type .llcrypt unregistered [ 7595.205692] Key type ._llcrypt unregistered [ 7613.957682] Key type ._llcrypt registered [ 7613.960372] Key type .llcrypt registered [ 7614.073434] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 7615.090777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7615.119082] alg: No test for adler32 (adler32-zlib) [ 7616.203621] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 7616.458054] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 7618.207234] Key type lgssc registered [ 7619.339109] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7626.741508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7626.757691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7628.278205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7633.259840] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7638.501468] LustreError: 191792:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7638.520651] LustreError: 191792:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7643.190855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7643.668644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7648.338264] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7655.170494] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7661.526913] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7669.818494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7670.020839] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7670.023188] LustreError: 193656:0:(llog.c:1696:llog_backup()) MGC192.168.201.118@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7670.030604] Lustre: 193656:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.118@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7670.146160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7676.960530] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7685.207277] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7688.622379] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7688.625840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7688.652926] Lustre: Skipped 2 previous similar messages [ 7688.685694] Lustre: Skipped 2 previous similar messages [ 7693.800967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7693.805591] Lustre: Skipped 2 previous similar messages [ 7694.232478] Lustre: server umount lustre-MDT0000 complete [ 7698.400759] LustreError: 191772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786634803 with bad export cookie 3586478396367811538 [ 7698.406954] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7698.416318] LustreError: 191772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7698.912182] LustreError: 191787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7698.912828] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7698.938759] LustreError: 192522:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000379bdb2b x1873422289234816/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7699.096902] Lustre: server umount lustre-MDT0001 complete [ 7705.979597] Lustre: server umount lustre-OST0000 complete [ 7717.845962] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 7728.054190] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 7763.998547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7764.221606] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7764.271234] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7764.379809] Lustre: lustre-MDT0000: new disk, initializing [ 7764.564447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7764.587844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7770.148813] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7781.210954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7781.335285] Lustre: 197181:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7781.403907] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7781.409413] Lustre: Skipped 1 previous similar message [ 7781.416502] LustreError: 197963:0:(llog.c:1696:llog_backup()) MGC192.168.201.118@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7781.426849] LustreError: 197963:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7781.432498] Lustre: 197963:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.118@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7781.451356] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7781.464862] Lustre: Skipped 1 previous similar message [ 7781.603944] Lustre: lustre-MDT0001: new disk, initializing [ 7781.719687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7781.788676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7781.803967] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7786.367140] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7795.965990] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7803.088900] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7810.608621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7810.957696] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7810.961625] Lustre: Skipped 1 previous similar message [ 7810.964650] LustreError: 199126:0:(llog.c:1696:llog_backup()) MGC192.168.201.118@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7810.972246] LustreError: 199126:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7810.975778] Lustre: 199126:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.118@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7810.982221] Lustre: 199126:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7810.998852] Lustre: lustre-OST0000: new disk, initializing [ 7811.006455] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7811.016370] Lustre: 199150:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7811.166731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7816.784612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7816.798771] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7816.959129] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7817.609273] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7827.180218] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7833.129868] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7833.323238] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7838.173393] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7838.399697] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7842.784691] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7842.791193] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7842.808549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7842.819342] Lustre: Skipped 1 previous similar message [ 7846.456607] Lustre: server umount lustre-OST0000 complete [ 7853.024659] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7853.036763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7853.052238] Lustre: Skipped 1 previous similar message [ 7853.068188] Lustre: Skipped 1 previous similar message [ 7858.151436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7858.163362] Lustre: Skipped 2 previous similar messages [ 7864.289223] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7864.593427] Lustre: server umount lustre-MDT0000 complete [ 7867.361384] LustreError: 197189:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7867.388923] LustreError: 197189:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7869.007616] LustreError: 199149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786634973 with bad export cookie 3586478396367812252 [ 7869.010148] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7869.018645] LustreError: 199149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7869.468502] Lustre: server umount lustre-MDT0001 complete [ 7878.083318] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 11:29:41 (1786634981) [ 7894.529413] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 7905.639993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7906.053288] LustreError: 202279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7906.105860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7910.806136] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7911.395259] LustreError: 202280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7920.467560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7925.703781] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7928.594643] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7936.337670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7943.502742] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7954.013209] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7954.310436] Lustre: lustre-OST0001: new disk, initializing [ 7954.314099] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7954.322571] Lustre: 204588:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7954.417117] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7954.425971] Lustre: Skipped 2 previous similar messages [ 7959.668947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7959.695668] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7959.840859] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7962.624258] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7975.048399] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7982.591723] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7984.901490] Lustre: DEBUG MARKER: ADD OST3 [ 7990.592554] loop: module loaded [ 7990.708343] loop0: detected capacity change from 0 to 204800000 [ 7991.037688] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7991.062032] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7991.076345] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7998.234874] loop0: detected capacity change from 0 to 204800000 [ 7998.772219] Lustre: DEBUG MARKER: STOP OST3 [ 8005.821916] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8006.055288] Lustre: lustre-OST0003: new disk, initializing [ 8006.061164] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8006.069095] Lustre: 206659:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8007.945810] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8008.201130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8008.220259] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8008.263390] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8009.172372] Lustre: Failing over lustre-OST0003 [ 8009.532969] Lustre: server umount lustre-OST0003 complete [ 8009.953094] LustreError: 206783:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8013.021735] LustreError: 204836:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8013.036770] LustreError: 204836:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8013.299768] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8014.318788] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8026.671941] Lustre: DEBUG MARKER: START OST3 again [ 8033.419188] LustreError: 203770:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8033.441223] LustreError: 203770:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 8034.566992] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8034.933534] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8034.946810] Lustre: Skipped 1 previous similar message [ 8034.961108] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8036.582596] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8036.681810] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8036.682711] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8042.228914] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8064.125141] Lustre: Failing over lustre-OST0003 [ 8064.329898] Lustre: server umount lustre-OST0003 complete [ 8066.038980] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8066.057562] Lustre: Skipped 1 previous similar message [ 8066.063844] LustreError: 203769:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8066.082780] LustreError: 203769:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8084.960804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8084.972599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8084.979928] Lustre: Skipped 1 previous similar message [ 8084.982683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8084.985376] Lustre: Skipped 2 previous similar messages [ 8089.074939] Lustre: server umount lustre-MDT0000 complete [ 8093.338944] LustreError: 202260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786635198 with bad export cookie 3586478396367813141 [ 8093.342073] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8093.353223] LustreError: 202260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8099.996838] Lustre: server umount lustre-MDT0001 complete [ 8105.374573] Lustre: server umount lustre-OST0000 complete [ 8109.947618] Lustre: server umount lustre-OST0001 complete [ 8117.547972] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 8129.602961] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 8172.085337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8172.337496] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8172.365587] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8172.444534] Lustre: lustre-MDT0000: new disk, initializing [ 8172.540563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8172.560889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8176.939949] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8188.581566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8188.733996] Lustre: 212200:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8188.791822] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8188.798045] Lustre: Skipped 1 previous similar message [ 8188.869971] Lustre: lustre-MDT0001: new disk, initializing [ 8189.045856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8189.072846] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8194.667870] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8205.950925] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8213.453409] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8222.311304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8222.740990] Lustre: lustre-OST0000: new disk, initializing [ 8222.744331] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8222.749575] Lustre: 214172:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8224.212401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8224.242359] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8224.503327] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8231.785263] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8241.908708] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8247.381870] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8247.682120] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8252.755522] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8252.965764] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8255.461051] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8255.472518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8255.484053] Lustre: Skipped 6 previous similar messages [ 8255.501608] Lustre: Skipped 9 previous similar messages [ 8261.465451] Lustre: server umount lustre-OST0000 complete [ 8272.021233] Lustre: server umount lustre-MDT0000 complete [ 8275.941307] LustreError: 212214:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8275.965768] LustreError: 212214:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 8277.168879] LustreError: 212194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786635382 with bad export cookie 3586478396367815437 [ 8277.178148] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8277.181405] LustreError: 212194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8277.569445] Lustre: server umount lustre-MDT0001 complete [ 8282.029394] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 11:36:24 (1786635384) [ 8303.234265] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 8315.934710] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 8356.950792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8357.195031] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8357.222481] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8357.359908] Lustre: lustre-MDT0000: new disk, initializing [ 8357.511435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8362.560371] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8374.039661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8374.169916] Lustre: 218782:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8374.398589] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8379.541288] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8389.680373] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8395.287734] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8403.097857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8403.502593] Lustre: 220753:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8405.472839] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8410.331335] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8420.222576] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8425.406588] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8425.717762] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8431.378790] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8431.646822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8436.202179] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8436.207671] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8436.251748] Lustre: Skipped 2 previous similar messages [ 8436.270625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8436.284534] Lustre: Skipped 7 previous similar messages [ 8440.620232] Lustre: server umount lustre-OST0000 complete [ 8445.920480] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8445.926545] LustreError: Skipped 1 previous similar message [ 8450.843762] Lustre: server umount lustre-MDT0000 complete [ 8451.564528] LustreError: 218796:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8451.591876] LustreError: 218796:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8455.004397] LustreError: 218776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786635559 with bad export cookie 3586478396367816319 [ 8455.016480] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8455.502719] Lustre: server umount lustre-MDT0001 complete [ 8466.323051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8467.259264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8467.270522] Lustre: Skipped 5 previous similar messages [ 8472.305071] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8482.577913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8490.107052] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8500.039395] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8507.391525] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8515.441471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8522.974733] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8531.043504] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8547.811649] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8547.824353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8547.829320] Lustre: Skipped 4 previous similar messages [ 8553.743881] Lustre: server umount lustre-OST0000 complete [ 8559.586799] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8559.609555] LustreError: Skipped 1 previous similar message [ 8564.597358] Lustre: server umount lustre-MDT0000 complete [ 8570.028478] LustreError: 222952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786635674 with bad export cookie 3586478396367817201 [ 8570.035512] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8570.042607] LustreError: 222952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8570.481475] Lustre: server umount lustre-MDT0001 complete [ 8669.962083] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 8672.605575] Key type lgssc unregistered [ 8672.931602] LNet: 226668:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8672.941569] LNetError: 226668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8672.952502] LNet: Removed LNI 192.168.201.118@tcp [ 8673.806215] Key type .llcrypt unregistered [ 8673.811316] Key type ._llcrypt unregistered [ 8703.412077] Key type ._llcrypt registered [ 8703.413504] Key type .llcrypt registered [ 8703.523423] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 8719.923184] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 8721.443528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8721.479591] alg: No test for adler32 (adler32-zlib) [ 8722.553482] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 8722.739569] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 8724.417745] Key type lgssc registered [ 8725.538879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8762.387691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8762.421073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8763.697370] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8763.726954] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8763.810763] Lustre: lustre-MDT0000: new disk, initializing [ 8763.902872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8763.924225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8768.964831] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8779.857331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8779.999813] Lustre: 230022:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8780.058822] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8780.062676] Lustre: Skipped 1 previous similar message [ 8780.140597] Lustre: lustre-MDT0001: new disk, initializing [ 8780.244679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8780.290737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8780.307724] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8785.054413] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8794.668037] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8800.035768] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8806.700948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8807.045521] Lustre: lustre-OST0000: new disk, initializing [ 8807.055195] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8807.069587] Lustre: 231986:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8807.197897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8813.347347] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8813.645737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8813.663265] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8813.788311] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8823.118228] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8828.019534] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8828.277987] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8833.179037] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8833.449907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8839.653645] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8839.664161] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8839.682578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8839.699355] Lustre: Skipped 1 previous similar message [ 8841.869505] Lustre: server umount lustre-OST0000 complete [ 8846.816647] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8846.838758] LustreError: Skipped 1 previous similar message [ 8846.845601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8846.858424] Lustre: Skipped 1 previous similar message [ 8846.874031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8849.890893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8849.893378] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8855.015828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8855.021290] Lustre: Skipped 2 previous similar messages [ 8859.114070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8859.123173] Lustre: Skipped 1 previous similar message [ 8860.131152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8860.501192] Lustre: server umount lustre-MDT0000 complete [ 8864.806591] LustreError: 230014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786635969 with bad export cookie 7943227267350516506 [ 8864.813602] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8864.821125] LustreError: 228245:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000848c913b x1873423448971904/t0(0) o41->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 224/224 e 0 to 0 dl 1786635967 ref 1 fl Rpc:RESQU/200/0 rc -5/-107 job:'' uid:0 gid:0 projid:4294967295 [ 8864.823356] LustreError: 230014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8865.198125] Lustre: server umount lustre-MDT0001 complete [ 8886.527706] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 8898.285183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8898.727495] LustreError: 234927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8898.743760] LustreError: 234927:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8898.797990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8903.956498] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8904.173367] LustreError: 234928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8908.260839] LustreError: 234927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8913.389550] LustreError: 234928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8914.544704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8915.025453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8921.149866] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8924.081986] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8932.225153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8932.762942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8939.444184] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8949.017266] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8949.260743] Lustre: lustre-OST0001: new disk, initializing [ 8949.265400] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8949.272482] Lustre: 237237:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8949.363869] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8955.479451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8955.494889] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8955.615519] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8956.476771] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8968.241922] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8977.949275] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8993.365591] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 11:48:16 (1786636096) [ 8996.839183] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8996.844553] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8996.847834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8996.875854] Lustre: Skipped 2 previous similar messages [ 9001.852588] Lustre: server umount lustre-MDT0000 complete [ 9001.954181] LustreError: 234928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9001.978847] LustreError: 234928:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9005.579444] LustreError: 234909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786636110 with bad export cookie 7943227267350517395 [ 9005.593336] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9005.599093] LustreError: 234909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9005.983851] Lustre: server umount lustre-MDT0001 complete [ 9020.573862] Lustre: server umount lustre-OST0000 complete [ 9022.432607] Lustre: 228246:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786636111/real 1786636111] req@ffff90c991e14380 x1873423449048832/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786636127 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9022.476477] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9022.496751] Lustre: Skipped 1 previous similar message [ 9024.368649] Lustre: server umount lustre-OST0001 complete [ 9030.526896] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 9041.740253] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9080.205953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9080.446269] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9080.467081] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9080.552788] Lustre: lustre-MDT0000: new disk, initializing [ 9080.642797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9080.662762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9085.411374] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9097.411795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9097.531498] Lustre: 241180:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9097.602377] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9097.606251] Lustre: Skipped 1 previous similar message [ 9097.687421] Lustre: lustre-MDT0001: new disk, initializing [ 9097.825667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9097.838279] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9102.363221] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9113.168986] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9119.089948] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9126.364291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9126.611196] Lustre: lustre-OST0000: new disk, initializing [ 9126.615965] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9126.622859] Lustre: 243148:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9128.053640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9128.067362] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9128.212286] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9132.624913] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9142.148036] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9147.497548] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9147.793902] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9154.209431] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9154.527483] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9159.137661] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9159.139329] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9159.155302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9159.155310] Lustre: Skipped 3 previous similar messages [ 9159.179398] LustreError: Skipped 1 previous similar message [ 9163.464354] Lustre: server umount lustre-OST0000 complete [ 9169.398664] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9174.503544] LustreError: 242074:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9174.923443] Lustre: server umount lustre-MDT0000 complete [ 9178.795187] LustreError: 241170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786636283 with bad export cookie 7943227267350519530 [ 9178.805101] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9178.808824] LustreError: 241170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9179.128964] Lustre: server umount lustre-MDT0001 complete [ 9188.039091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9188.510664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9188.520856] Lustre: Skipped 2 previous similar messages [ 9193.528741] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9193.966220] LustreError: 245364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9193.996351] LustreError: 245364:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9202.514250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9207.310367] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9214.885257] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9221.234678] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9229.126946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9237.169765] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9246.352766] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9393.633310] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9393.650877] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9393.666360] Lustre: Skipped 3 previous similar messages [ 9393.672511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9393.680146] Lustre: Skipped 3 previous similar messages [ 9398.530510] Lustre: server umount lustre-OST0000 complete [ 9409.984377] Lustre: server umount lustre-MDT0000 complete [ 9412.068319] LustreError: 246759:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9412.088719] LustreError: 246759:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9413.643246] LustreError: 245345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786636518 with bad export cookie 7943227267350520412 [ 9413.645297] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9413.652051] LustreError: 245345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9413.905449] Lustre: server umount lustre-MDT0001 complete [ 9514.637860] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 9517.309304] Key type lgssc unregistered [ 9517.650871] LNet: 248977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9517.656654] LNetError: 248977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9517.669303] LNet: Removed LNI 192.168.201.118@tcp [ 9518.669303] Key type .llcrypt unregistered [ 9518.673342] Key type ._llcrypt unregistered [ 9544.490824] Key type ._llcrypt registered [ 9544.493673] Key type .llcrypt registered [ 9544.600163] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 9560.674204] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9562.332590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9562.502303] alg: No test for adler32 (adler32-zlib) [ 9563.753767] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 9564.103425] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 9565.823512] Key type lgssc registered [ 9567.317644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9616.554103] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9626.526959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9626.565083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9627.795548] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9627.818125] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9627.897192] Lustre: lustre-MDT0000: new disk, initializing [ 9627.975568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9627.998673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9632.853923] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9644.245432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9644.365246] Lustre: 253050:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9644.421271] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9644.425458] Lustre: Skipped 1 previous similar message [ 9644.501997] Lustre: lustre-MDT0001: new disk, initializing [ 9644.589605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9644.643491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9644.663408] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9648.995116] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9653.949365] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9661.377262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9661.699831] Lustre: lustre-OST0000: new disk, initializing [ 9661.706516] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9661.717185] Lustre: 254680:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9661.827434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9666.652786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9666.669241] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9666.790066] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9668.730413] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9679.865743] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9680.039690] Lustre: lustre-OST0001: new disk, initializing [ 9680.044535] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9680.050946] Lustre: 255550:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9680.185968] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9687.739485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9687.760915] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9687.815030] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9687.872157] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9699.405131] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9706.698412] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9723.879295] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9723.881140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9723.892566] Lustre: Skipped 2 previous similar messages [ 9723.895507] Lustre: Skipped 1 previous similar message [ 9728.379902] Lustre: server umount lustre-MDT0000 complete [ 9733.174230] LustreError: 253042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786636838 with bad export cookie 17440371818457892868 [ 9733.174942] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9733.184638] LustreError: 253042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9733.474239] Lustre: server umount lustre-MDT0001 complete [ 9748.490751] Lustre: server umount lustre-OST0000 complete [ 9750.496710] Lustre: 250558:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786636839/real 1786636839] req@ffff90c988059500 x1873424331911808/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786636855 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9750.539883] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9750.555564] Lustre: Skipped 1 previous similar message [ 9753.525937] Lustre: server umount lustre-OST0001 complete [ 9768.650441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9780.521401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9790.571671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9800.584082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9822.956212] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9834.138414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9834.167672] Lustre: lustre-MDT0000: reset Object Index mappings [ 9834.567414] LustreError: 260572:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9834.590543] LustreError: 260572:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 9834.653829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9839.656118] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9840.112153] LustreError: 260573:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9844.195301] LustreError: 260572:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9848.956800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9848.993081] Lustre: lustre-MDT0001: reset Object Index mappings [ 9849.396334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9854.544163] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9857.880458] Lustre: 261717:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9865.513244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9865.911482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9872.498244] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9874.097692] LustreError: 262070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9879.210860] LustreError: 262069:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9879.234710] LustreError: 262069:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9879.239983] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9880.854472] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9886.205190] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9887.548473] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9895.161796] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9898.440163] Lustre: 263586:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9905.650295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9905.656414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9905.669661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9905.678052] Lustre: Skipped 2 previous similar messages [ 9908.870856] Lustre: server umount lustre-MDT0000 complete [ 9911.783432] LustreError: 260567:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9911.814363] LustreError: 260567:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9912.172876] LustreError: 260554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786637017 with bad export cookie 17440371818457899658 [ 9912.180973] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9912.182626] LustreError: 260554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9912.451427] Lustre: server umount lustre-MDT0001 complete [ 9926.215179] Lustre: server umount lustre-OST0000 complete [ 9939.919162] Lustre: server umount lustre-OST0001 complete [ 9959.344862] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 12:04:22 (1786637062) [ 9980.691428] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 9994.229285] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [10034.855531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10035.055870] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10035.080313] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10035.134991] Lustre: lustre-MDT0000: new disk, initializing [10035.204362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10035.210037] Lustre: Skipped 1 previous similar message [10035.224194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10038.920780] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10048.693213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10048.814387] Lustre: 267844:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10048.860644] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10048.864063] Lustre: Skipped 1 previous similar message [10048.918549] Lustre: lustre-MDT0001: new disk, initializing [10049.027668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10049.047989] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10053.107567] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10062.354609] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10067.110904] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10073.725102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10073.989480] Lustre: lustre-OST0000: new disk, initializing [10073.997322] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10074.005892] Lustre: 269811:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10075.487130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10075.508445] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10075.673169] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10081.079743] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10090.588142] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10095.050854] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10095.273890] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10099.909431] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10100.090449] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10106.338590] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10106.341592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10106.343733] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10106.351535] Lustre: Skipped 4 previous similar messages [10106.360404] Lustre: Skipped 5 previous similar messages [10107.827658] Lustre: server umount lustre-OST0000 complete [10111.457589] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10111.460620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10111.471048] Lustre: Skipped 1 previous similar message [10116.577337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10116.582527] Lustre: Skipped 2 previous similar messages [10117.496583] Lustre: server umount lustre-MDT0000 complete [10120.602659] LustreError: 267835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786637225 with bad export cookie 17440371818457901744 [10120.608463] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10120.610637] LustreError: 267835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10120.913517] Lustre: server umount lustre-MDT0001 complete [10136.958896] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [10146.326946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10146.617723] LustreError: 272744:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10146.663427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10146.668375] Lustre: Skipped 2 previous similar messages [10150.602150] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10157.243718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10161.155495] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10163.712310] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10169.945745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10176.243620] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10184.543412] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10184.748790] Lustre: lustre-OST0001: new disk, initializing [10184.752946] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10184.766409] Lustre: 275051:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10186.623398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10186.628220] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10186.717345] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10191.769641] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10200.408758] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10203.135442] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10207.928079] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10207.959830] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10213.857285] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10213.869530] LustreError: Skipped 1 previous similar message [10213.879128] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10213.892193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10214.085513] Lustre: server umount lustre-MDT0000 complete [10217.448418] LustreError: 272744:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10217.471568] LustreError: 272744:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10217.516350] LustreError: 272725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786637322 with bad export cookie 17440371818457902626 [10217.521868] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10217.533024] LustreError: 272725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10219.910230] Lustre: server umount lustre-MDT0001 complete [10234.802601] Lustre: server umount lustre-OST0000 complete [10248.606348] Lustre: server umount lustre-OST0001 complete [10264.487578] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [10273.861672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10279.540733] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10284.516903] LustreError: 278108:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10284.527215] LustreError: 278108:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10287.901865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10288.355266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10288.365168] Lustre: Skipped 4 previous similar messages [10292.885770] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10295.812124] Lustre: 279247:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10302.197497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10309.158441] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10317.477524] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10319.811706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10319.892484] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10324.741107] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10332.132252] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10335.889614] Lustre: 281124:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10340.735360] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10340.822833] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10349.830485] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 12:10:53 (1786637453) [10365.935514] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10365.945943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10365.955708] Lustre: Skipped 6 previous similar messages [10365.963938] Lustre: Skipped 2 previous similar messages [10371.384272] Lustre: server umount lustre-MDT0000 complete [10374.798993] LustreError: 278088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786637479 with bad export cookie 17440371818457908947 [10374.819317] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10374.819357] LustreError: 278088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10381.210721] Lustre: server umount lustre-MDT0001 complete [10391.599579] Lustre: server umount lustre-OST0000 complete [10401.550828] Lustre: server umount lustre-OST0001 complete [10408.386640] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [10418.869458] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [10468.180988] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [10478.858171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10479.191401] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10479.232876] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10479.373815] Lustre: lustre-MDT0000: new disk, initializing [10479.516675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10484.601549] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10496.787745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10496.958409] Lustre: 285368:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10497.404728] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10502.771521] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10515.884764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10516.233583] Lustre: 286998:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10518.488958] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10524.287400] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10534.477165] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10534.627859] Lustre: 287868:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10536.250857] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10536.263515] Lustre: Skipped 1 previous similar message [10536.341986] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10540.693677] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10550.386618] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10553.889747] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10553.894038] Lustre: Skipped 1 previous similar message [10567.809683] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 12:14:31 (1786637671) [10572.260777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10572.263374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10572.283148] Lustre: Skipped 5 previous similar messages [10572.286635] Lustre: Skipped 11 previous similar messages [10575.970433] Lustre: server umount lustre-MDT0000 complete [10577.391936] LustreError: 285379:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10577.454396] LustreError: 285379:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [10579.766515] LustreError: 285361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786637684 with bad export cookie 17440371818457916815 [10579.767883] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10579.779732] LustreError: 285361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10580.172899] Lustre: server umount lustre-MDT0001 complete [10593.381282] Lustre: server umount lustre-OST0000 complete [10608.034508] Lustre: server umount lustre-OST0001 complete [10616.058662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10616.640750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10616.648631] Lustre: Skipped 6 previous similar messages [10622.229674] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10631.528356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10636.967398] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10643.963347] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10649.186963] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10655.182649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10661.466455] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10669.372375] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10685.618157] loop1: detected capacity change from 0 to 204800000 [10685.944278] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10685.974491] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10685.996215] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10692.144482] loop1: detected capacity change from 0 to 204800000 [10697.293383] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10697.408318] LustreError: 290108:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs1113: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10697.424786] LustreError: 290108:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs1113 [10697.430398] LustreError: 290108:0:(mgs_handler.c:517:mgs_target_reg()) fs1113-MDT0001: Error checking target index: rc = -13 [10697.439490] LustreError: fs1113-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10697.452485] LustreError: 293834:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10697.460324] LustreError: 293834:0:(tgt_mount.c:2129:server_put_super()) no obd fs1113-MDT0001 [10697.469930] LustreError: 293834:0:(tgt_mount.c:116:server_deregister_mount()) fs1113-MDT0001 not registered [10697.517313] Lustre: server umount fs1113-MDT0001 complete [10697.525847] LustreError: 293834:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10706.562262] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10706.739993] Lustre: Setting parameter fs1113-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs1113-MDT0001 [10706.790430] Lustre: srv-fs1113-MDT0001: No data found on store. Initialize space. [10706.799297] Lustre: Skipped 4 previous similar messages [10706.916384] Lustre: fs1113-MDT0001: new disk, initializing [10706.919786] Lustre: Skipped 3 previous similar messages [10707.083670] Lustre: srv-fs1113-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10707.095967] Lustre: 294301:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1113-MDT0001-mdtlov: can't create new llog: rc = -115 [10707.103415] LustreError: 294301:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1113-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10711.642646] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10718.202420] Lustre: server umount fs1113-MDT0001 complete [10730.028910] loop1: detected capacity change from 0 to 204800000 [10736.560645] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10736.721602] Lustre: 295870:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1113-MDT0001-mdtlov: can't create new llog: rc = -115 [10736.727083] LustreError: 295870:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1113-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10740.065400] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10743.795243] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10745.608585] Lustre: server umount fs1113-MDT0001 complete [10759.153899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10759.169246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10759.179326] Lustre: Skipped 2 previous similar messages [10759.215298] Lustre: Skipped 3 previous similar messages [10764.054298] Lustre: server umount lustre-MDT0000 complete [10771.959192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10772.083931] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10776.846947] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10777.600754] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10783.912692] loop1: detected capacity change from 0 to 204800000 [10790.031089] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10790.759533] Lustre: 298316:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1113-MDT0001-mdtlov: can't create new llog: rc = -115 [10790.770353] LustreError: 298316:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1113-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10794.604568] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10798.863099] Lustre: server umount fs1113-MDT0001 complete [10818.532591] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10818.540409] LustreError: Skipped 1 previous similar message [10832.351383] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10832.794322] Lustre: server umount lustre-OST0000 complete [10839.009700] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10845.471652] Lustre: server umount lustre-MDT0000 complete [10849.254643] LustreError: 290114:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10849.275788] LustreError: 290114:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 106 previous similar messages [10850.467696] LustreError: 298808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786637955 with bad export cookie 17440371818457919496 [10850.483748] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10851.125815] Lustre: server umount lustre-MDT0001 complete [10866.088172] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [10868.192155] Key type lgssc unregistered [10868.708127] LNet: 300665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10868.721943] LNetError: 300665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10868.771895] LNet: Removed LNI 192.168.201.118@tcp [10869.590268] Key type .llcrypt unregistered [10869.593381] Key type ._llcrypt unregistered [10887.829257] Key type ._llcrypt registered [10887.831257] Key type .llcrypt registered [10887.907175] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [10889.061444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10889.118149] alg: No test for adler32 (adler32-zlib) [10890.249551] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [10890.465850] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [10892.127220] Key type lgssc registered [10893.420592] Lustre: Echo OBD driver; http://www.lustre.org/ [10900.784733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10900.801540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10902.241632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10906.471934] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10912.740339] LustreError: 301918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10912.778683] LustreError: 301918:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10914.455361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10914.879544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10919.321431] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10928.006901] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10934.337414] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10941.187450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10941.483177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10947.628619] LustreError: 303751:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10949.635224] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10952.677802] LustreError: 303751:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10957.803779] LustreError: 303753:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10958.125241] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10964.965898] LustreError: 304035:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10965.004428] LustreError: 304035:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10975.213522] LustreError: 303753:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10975.223925] LustreError: 303753:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10980.131841] loop1: detected capacity change from 0 to 204800000 [10980.402133] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10980.429743] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10980.452037] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10986.409795] loop1: detected capacity change from 0 to 204800000 [10991.759732] LustreError: 304035:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10991.791619] LustreError: 304035:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [10993.242590] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10993.475281] Lustre: fs27315-OST0000: new disk, initializing [10993.484525] Lustre: srv-fs27315-OST0000: No data found on store. Initialize space. [10993.493251] Lustre: 305655:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10993.583814] Lustre: fs27315-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11003.852797] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11014.882715] Lustre: server umount fs27315-OST0000 complete [11026.431112] LustreError: 303751:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11026.457887] LustreError: 303751:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [11026.496467] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11031.697674] loop1: detected capacity change from 0 to 204800000 [11039.407837] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11039.825063] Lustre: fs27315-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11041.765348] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11048.048884] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11058.017039] loop2: detected capacity change from 0 to 204800000 [11058.291497] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11058.306282] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11058.334677] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11063.741522] loop2: detected capacity change from 0 to 204800000 [11069.651972] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11069.787959] LustreError: fs27315-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11069.801785] LustreError: fs27315-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11069.816964] LustreError: 308698:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11069.832364] LustreError: 308698:0:(tgt_mount.c:2129:server_put_super()) no obd fs27315-OST0001 [11069.838441] LustreError: 308698:0:(tgt_mount.c:116:server_deregister_mount()) fs27315-OST0001 not registered [11069.880343] Lustre: server umount fs27315-OST0001 complete [11069.885240] LustreError: 308698:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11080.220673] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11080.447708] Lustre: fs27315-OST0001: new disk, initializing [11080.454500] Lustre: srv-fs27315-OST0001: No data found on store. Initialize space. [11080.462610] Lustre: 309248:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11080.572604] Lustre: fs27315-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11088.439599] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11092.971032] LustreError: 303752:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11093.002539] LustreError: 303752:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [11099.630476] Lustre: server umount fs27315-OST0000 complete [11109.438503] Lustre: server umount fs27315-OST0001 complete [11124.655382] loop1: detected capacity change from 0 to 204800000 [11130.955347] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11131.353128] Lustre: fs27315-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11139.827364] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11147.269920] loop2: detected capacity change from 0 to 204800000 [11153.363442] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11160.263720] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11170.762478] Lustre: server umount fs27315-OST0000 complete [11179.414822] Lustre: server umount fs27315-OST0001 complete [11180.009681] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11195.362768] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11195.365202] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11195.370143] LustreError: Skipped 1 previous similar message [11195.372636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11195.401374] Lustre: Skipped 1 previous similar message [11200.494312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11200.508875] Lustre: Skipped 1 previous similar message [11205.606934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11205.617996] Lustre: Skipped 1 previous similar message [11208.161326] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11208.491864] Lustre: server umount lustre-OST0000 complete [11215.844109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11215.860104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11215.871068] Lustre: Skipped 1 previous similar message [11218.996189] Lustre: server umount lustre-MDT0000 complete [11223.035768] LustreError: 301900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786638327 with bad export cookie 760159143626543068 [11223.046317] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11223.056139] LustreError: 301900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11223.368506] Lustre: server umount lustre-MDT0001 complete [11235.525902] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [11238.145582] Key type lgssc unregistered [11238.423792] LNet: 315181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11238.431648] LNetError: 315181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11238.446385] LNet: Removed LNI 192.168.201.118@tcp [11239.191177] Key type .llcrypt unregistered [11239.194906] Key type ._llcrypt unregistered [11274.734906] Key type ._llcrypt registered [11274.737296] Key type .llcrypt registered [11274.855533] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [11291.550819] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [11293.422415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11293.531395] alg: No test for adler32 (adler32-zlib) [11294.657345] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [11295.046864] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [11296.751238] Key type lgssc registered [11298.763790] Lustre: Echo OBD driver; http://www.lustre.org/ [11342.174039] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11342.249891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11343.594523] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11343.637535] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11343.770976] Lustre: lustre-MDT0000: new disk, initializing [11343.872223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11343.896726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11349.671654] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11361.398200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11361.545294] Lustre: 319044:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11361.614721] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11361.622976] Lustre: Skipped 1 previous similar message [11361.754454] Lustre: lustre-MDT0001: new disk, initializing [11361.915965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11361.992988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11362.027884] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11367.254742] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11378.209563] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11384.936815] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11393.247285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11393.785969] Lustre: lustre-OST0000: new disk, initializing [11393.790098] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11393.799291] Lustre: 321014:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11393.931695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11399.323526] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11399.704599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11399.716915] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11399.808535] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11406.886611] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11408.912827] Lustre: Mounted lustre-client - version 2.17.57_1_gb6c95b3 [11409.695730] Lustre: Unmounted lustre-client [11419.113213] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11419.115236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11419.140522] Lustre: Skipped 1 previous similar message [11419.155117] Lustre: Skipped 1 previous similar message [11424.184372] Lustre: server umount lustre-OST0000 complete [11428.326711] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11428.338880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11428.369697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11429.362465] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11429.385083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11429.410803] Lustre: Skipped 1 previous similar message [11434.393130] Lustre: server umount lustre-MDT0000 complete [11437.658151] LustreError: 319035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786638542 with bad export cookie 13627266680306532837 [11437.659937] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11437.675308] LustreError: 319035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11438.017505] Lustre: server umount lustre-MDT0001 complete [11447.216785] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [11449.534977] Key type lgssc unregistered [11464.033162] LNet: 1 local NIs in recovery (showing 1): 192.168.201.118@tcp [11464.300389] LNet: 323176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11464.315779] LNetError: 323176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11540.141846] LNet: Removed LNI 192.168.201.118@tcp [11541.043918] Key type .llcrypt unregistered [11541.046847] Key type ._llcrypt unregistered [11558.883116] Key type ._llcrypt registered [11558.885287] Key type .llcrypt registered [11558.990306] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [11573.842760] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [11574.961418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11575.051262] alg: No test for adler32 (adler32-zlib) [11576.074976] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [11576.307945] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [11577.991326] Key type lgssc registered [11579.052324] Lustre: Echo OBD driver; http://www.lustre.org/ [11607.053858] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 12:31:50 (1786638710) [11612.771815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11612.812735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11614.016707] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11614.037485] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11614.105618] Lustre: lustre-MDT0000: new disk, initializing [11614.148582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11614.157607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11617.507491] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11626.269659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11626.381145] Lustre: 326399:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11626.413989] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11626.423716] Lustre: Skipped 1 previous similar message [11626.527622] Lustre: lustre-MDT0001: new disk, initializing [11626.603818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11626.628658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11626.646540] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11630.280979] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11637.798423] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11642.501941] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11649.006935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11649.332837] Lustre: lustre-OST0000: new disk, initializing [11649.345682] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11649.352274] Lustre: 328366:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11649.439811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11655.210414] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11658.865023] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11658.883672] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11658.984154] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11664.552449] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11679.713322] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11679.714621] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11679.722247] LustreError: Skipped 1 previous similar message [11679.724851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11679.734525] Lustre: Skipped 1 previous similar message [11679.747574] Lustre: Skipped 1 previous similar message [11684.840133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11684.852017] Lustre: Skipped 1 previous similar message [11685.001591] Lustre: server umount lustre-OST0000 complete [11689.955950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11689.961770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11689.973056] Lustre: Skipped 1 previous similar message [11689.986057] Lustre: Skipped 1 previous similar message [11694.732517] Lustre: server umount lustre-MDT0000 complete [11697.882261] LustreError: 326391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786638802 with bad export cookie 13561445145546308166 [11697.882640] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11697.889294] LustreError: 326391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11698.155527] Lustre: server umount lustre-MDT0001 complete [11707.668813] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [11710.301271] Key type lgssc unregistered [11710.589145] LNet: 330277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11710.598480] LNetError: 330277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11710.621082] LNet: Removed LNI 192.168.201.118@tcp [11711.512561] Key type .llcrypt unregistered [11711.514684] Key type ._llcrypt unregistered [11732.682796] Key type ._llcrypt registered [11732.686692] Key type .llcrypt registered [11732.761712] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [11733.564859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11733.688675] alg: No test for adler32 (adler32-zlib) [11734.637358] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [11734.881339] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [11736.567183] Key type lgssc registered [11737.396090] Lustre: Echo OBD driver; http://www.lustre.org/ [11744.049669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11744.103192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11745.601547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11750.381412] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11756.011159] LustreError: 331552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11756.029494] LustreError: 331552:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11758.136249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11758.621281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11761.587448] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11768.768133] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11774.656973] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11782.871928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11783.214758] LustreError: 333372:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11783.544667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11790.481979] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11799.438436] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11809.169538] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11810.681284] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11814.881368] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11814.897719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11814.914056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11816.421528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11816.429669] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11816.458108] Lustre: Skipped 1 previous similar message [11817.638121] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.18@tcp (stopping) [11817.648392] Lustre: Skipped 2 previous similar messages [11821.540571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11821.559430] Lustre: Skipped 2 previous similar messages [11826.664151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11826.672364] Lustre: Skipped 3 previous similar messages [11829.215259] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11829.494609] Lustre: server umount lustre-MDT0000 complete [11831.784766] LustreError: 331547:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11831.814744] LustreError: 331547:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11833.446238] LustreError: 331533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786638938 with bad export cookie 11379468615362669139 [11833.449682] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11833.461122] LustreError: 331533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11836.905908] LustreError: 331547:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11836.908568] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11836.935949] LustreError: 331547:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11836.975908] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11836.989922] Lustre: Skipped 1 previous similar message [11839.903760] Lustre: server umount lustre-MDT0001 complete [11843.816987] Lustre: server umount lustre-OST0000 complete [11851.858523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11855.397903] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11858.195986] Lustre: 335733:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.118@tcp [11859.978083] Lustre: server umount MGS complete [11866.495677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11866.778250] LustreError: 336303:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11866.789976] LustreError: 336303:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11866.837657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11870.683592] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11876.330194] LustreError: 336304:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11876.353996] LustreError: 336304:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11877.813681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11878.127258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11882.028710] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11890.272057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11890.840736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11896.885430] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11903.768378] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11905.754308] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11911.601250] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11913.721235] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11919.864351] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [11919.880313] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11919.903530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11919.911383] Lustre: Skipped 2 previous similar messages [11920.322723] Lustre: server umount lustre-OST0000 complete [11924.449086] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11924.458820] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11938.143517] Lustre: 330863:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786639027/real 1786639027] req@000000001d8467cb x1873426607305600/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786639043 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11938.165651] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11938.187744] Lustre: Skipped 1 previous similar message [11938.271982] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11938.538311] Lustre: server umount lustre-MDT0000 complete [11941.858810] LustreError: 336284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786639046 with bad export cookie 11379468615362669979 [11941.873310] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11942.131850] Lustre: server umount lustre-MDT0001 complete [11953.377745] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [11956.275050] Key type lgssc unregistered [11956.575058] LNet: 339742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11956.581820] LNetError: 339742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11956.595958] LNet: Removed LNI 192.168.201.118@tcp [11957.554721] Key type .llcrypt unregistered [11957.556471] Key type ._llcrypt unregistered [11974.349267] Key type ._llcrypt registered [11974.351541] Key type .llcrypt registered [11974.457518] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [11988.295777] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [11989.184638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11989.239430] alg: No test for adler32 (adler32-zlib) [11990.232634] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [11990.420052] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [11992.087148] Key type lgssc registered [11993.196282] Lustre: Echo OBD driver; http://www.lustre.org/ [12026.917555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12026.956436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12028.227963] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12028.263947] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12028.329440] Lustre: lustre-MDT0000: new disk, initializing [12028.379840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12028.398750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12032.837236] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12042.374581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12042.514426] Lustre: 342903:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12042.561302] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12042.566940] Lustre: Skipped 1 previous similar message [12042.650679] Lustre: lustre-MDT0001: new disk, initializing [12042.773085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12042.830357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12042.841258] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12046.948577] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12055.176684] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12060.952918] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12067.812541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12068.090937] Lustre: lustre-OST0000: new disk, initializing [12068.094726] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12068.103191] Lustre: 344871:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12068.173919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12074.607492] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12078.225841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12078.242764] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12078.448303] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12085.207840] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12090.044691] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12090.313810] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12095.651961] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12095.880989] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12099.042470] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12099.055141] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12099.058537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12099.085414] LustreError: Skipped 1 previous similar message [12104.165972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12104.173831] Lustre: Skipped 2 previous similar messages [12109.298183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12112.354364] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12112.483249] Lustre: server umount lustre-OST0000 complete [12119.520519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12119.522934] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12119.529874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12119.537402] Lustre: Skipped 2 previous similar messages [12119.543745] Lustre: Skipped 2 previous similar messages [12122.270550] Lustre: server umount lustre-MDT0000 complete [12125.810395] LustreError: 342896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786639230 with bad export cookie 7333695687702129240 [12125.821785] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12125.823503] LustreError: 342896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12126.350292] Lustre: server umount lustre-MDT0001 complete [12136.004945] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 12:40:38 (1786639238) [12153.698168] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [12163.904626] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [12209.433842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12209.852112] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12209.890246] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12209.970339] Lustre: lustre-MDT0000: new disk, initializing [12210.071609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12210.093610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12215.215579] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12227.319090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12227.480074] Lustre: 349634:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12227.543401] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12227.549985] Lustre: Skipped 1 previous similar message [12227.666207] Lustre: lustre-MDT0001: new disk, initializing [12227.796599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12227.862818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12227.873027] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12232.148536] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12240.327752] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12245.591646] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12253.354757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12253.527512] Lustre: lustre-OST0000: new disk, initializing [12253.529879] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12253.535303] Lustre: 351599:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12253.588911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12259.652440] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12260.970418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12260.998167] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12261.250436] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12269.754647] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12307.426930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12307.434302] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12307.442472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12307.454715] Lustre: Skipped 1 previous similar message [12309.963720] Lustre: server umount lustre-OST0000 complete [12315.104643] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12315.110813] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12317.663907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12317.674955] Lustre: Skipped 3 previous similar messages [12320.100057] Lustre: server umount lustre-MDT0000 complete [12322.792625] LustreError: 349640:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12322.817114] LustreError: 349640:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12323.187468] LustreError: 349627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786639428 with bad export cookie 7333695687702130129 [12323.200933] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12323.487131] Lustre: server umount lustre-MDT0001 complete [12333.671665] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [12336.053346] Key type lgssc unregistered [12336.315070] LNet: 353779:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12336.320517] LNetError: 353779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12336.334834] LNet: Removed LNI 192.168.201.118@tcp [12336.995751] Key type .llcrypt unregistered [12336.997630] Key type ._llcrypt unregistered [12436.027420] Key type ._llcrypt registered [12436.029834] Key type .llcrypt registered [12436.185068] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [12436.516648] Key type .llcrypt unregistered [12436.521142] Key type ._llcrypt unregistered [12510.685263] Key type ._llcrypt registered [12510.686803] Key type .llcrypt registered [12510.796399] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [12511.127450] Key type .llcrypt unregistered [12511.136182] Key type ._llcrypt unregistered [12539.098596] Key type ._llcrypt registered [12539.100862] Key type .llcrypt registered [12539.194913] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [12539.466560] Key type .llcrypt unregistered [12539.469299] Key type ._llcrypt unregistered [12570.471640] Key type ._llcrypt registered [12570.475114] Key type .llcrypt registered [12570.594547] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [12570.915550] Key type .llcrypt unregistered [12570.918854] Key type ._llcrypt unregistered [12659.538665] Key type ._llcrypt registered [12659.541709] Key type .llcrypt registered [12659.621756] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [12659.838371] Key type .llcrypt unregistered [12659.839917] Key type ._llcrypt unregistered [12686.959798] Key type ._llcrypt registered [12686.961885] Key type .llcrypt registered [12687.031793] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [12687.265949] Key type .llcrypt unregistered [12687.268230] Key type ._llcrypt unregistered [12716.547657] Key type ._llcrypt registered [12716.553032] Key type .llcrypt registered [12716.676184] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [12717.119579] Key type .llcrypt unregistered [12717.126121] Key type ._llcrypt unregistered [12749.438246] Key type ._llcrypt registered [12749.439343] Key type .llcrypt registered [12749.490509] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [12749.670311] Key type .llcrypt unregistered [12749.677100] Key type ._llcrypt unregistered [12777.017627] Key type ._llcrypt registered [12777.019031] Key type .llcrypt registered [12777.140527] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [12777.390364] Key type .llcrypt unregistered [12777.392555] Key type ._llcrypt unregistered [12804.803508] Key type ._llcrypt registered [12804.809917] Key type .llcrypt registered [12804.955906] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [12805.288592] Key type .llcrypt unregistered [12805.296790] Key type ._llcrypt unregistered [12869.888902] Key type ._llcrypt registered [12869.890578] Key type .llcrypt registered [12870.022949] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [12870.507641] Key type .llcrypt unregistered [12870.522937] Key type ._llcrypt unregistered [12901.400489] Key type ._llcrypt registered [12901.404467] Key type .llcrypt registered [12901.515607] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [12901.780465] Key type .llcrypt unregistered [12901.784726] Key type ._llcrypt unregistered [12933.782973] Key type ._llcrypt registered [12933.788699] Key type .llcrypt registered [12933.925238] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [12934.185546] Key type .llcrypt unregistered [12934.187829] Key type ._llcrypt unregistered [12982.020981] Key type ._llcrypt registered [12982.023448] Key type .llcrypt registered [12982.098271] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [12998.993900] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [12999.745401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12999.791532] alg: No test for adler32 (adler32-zlib) [13000.753207] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [13001.011753] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [13002.751407] Key type lgssc registered [13003.795934] Lustre: Echo OBD driver; http://www.lustre.org/ [13040.426696] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13040.460872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13041.785257] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13041.810066] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13041.879649] Lustre: tst250fs-MDT0000: new disk, initializing [13041.971137] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13041.985615] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13047.606268] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [13058.243617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13058.376978] Lustre: 374148:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13058.434122] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13058.438505] Lustre: Skipped 1 previous similar message [13058.506822] Lustre: tst250fs-MDT0001: new disk, initializing [13058.606598] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13058.644518] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13058.659697] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13063.088565] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [13068.029822] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13074.644534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13074.993634] Lustre: tst250fs-OST0000: new disk, initializing [13075.005717] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13075.011884] Lustre: 375780:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13075.117530] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13082.211351] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13082.228266] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13082.278655] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [13082.392410] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13093.073298] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13093.373985] Lustre: tst250fs-OST0001: new disk, initializing [13093.379876] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13093.388880] Lustre: 376653:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13093.502375] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13099.873152] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [13101.289882] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13101.297377] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13101.503935] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13106.244698] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13108.895948] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13108.899969] Lustre: Skipped 2 previous similar messages [13127.150399] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13127.159298] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13127.162906] Lustre: Skipped 3 previous similar messages [13127.193961] Lustre: Skipped 3 previous similar messages [13132.320417] Lustre: server umount tst250fs-MDT0000 complete [13135.791690] LustreError: 374141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786640240 with bad export cookie 5333701037511277070 [13135.794074] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13135.806116] LustreError: 374141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13136.155349] Lustre: server umount tst250fs-MDT0001 complete [13150.471446] Lustre: server umount tst250fs-OST0000 complete [13153.762876] Lustre: 372371:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786640242/real 1786640242] req@ffff90c990c7ad80 x1873427934811136/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786640258 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13153.790900] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13153.958292] Lustre: server umount tst250fs-OST0001 complete [13172.556344] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [13181.760804] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [13216.051121] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 12:58:39 (1786640319) [13217.552743] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13233.522905] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [13244.039273] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [13276.189655] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12946 sec ========= 12:59:38 (1786640378) [13278.913161] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:59:41 (1786640381) === [13282.928255] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:59:45 (1786640385) === [13324.826876] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [13327.955781] Key type lgssc unregistered [13328.283038] LNet: 385796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13328.292371] LNetError: 385796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13328.317823] LNet: Removed LNI 192.168.201.118@tcp [13329.195358] Key type .llcrypt unregistered [13329.197082] Key type ._llcrypt unregistered