[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 1016249712 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003041] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005011] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009022] pid_max: default: 32768 minimum: 301 [ 0.011161] LSM: Security Framework initializing [ 0.012066] Yama: becoming mindful. [ 0.013046] SELinux: Initializing. [ 0.014069] *** VALIDATE selinux *** [ 0.022469] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026734] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027157] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028101] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029103] *** VALIDATE tmpfs *** [ 0.031391] *** VALIDATE proc *** [ 0.032216] *** VALIDATE cgroup *** [ 0.033007] *** VALIDATE cgroup2 *** [ 0.035187] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036145] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038029] Spectre V2 : User space: Vulnerable [ 0.039006] Speculative Store Bypass: Vulnerable [ 0.042234] debug: unmapping init [mem 0xffffffffac659000-0xffffffffac660fff] [ 0.044890] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045817] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046024] ... version: 2 [ 0.047014] ... bit width: 48 [ 0.048009] ... generic registers: 4 [ 0.049012] ... value mask: 0000ffffffffffff [ 0.050011] ... max period: 00007fffffffffff [ 0.051015] ... fixed-purpose events: 3 [ 0.052010] ... event mask: 000000070000000f [ 0.053278] rcu: Hierarchical SRCU implementation. [ 0.055784] smp: Bringing up secondary CPUs ... [ 0.056847] x86: Booting SMP configuration: [ 0.057019] .... node #0, CPUs: #1 #2 #3 [ 0.066060] smp: Brought up 1 node, 4 CPUs [ 0.068022] smpboot: Max logical packages: 1 [ 0.069017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.118165] node 0 deferred pages initialised in 47ms [ 0.122037] devtmpfs: initialized [ 0.123309] x86/mm: Memory block size: 128MB [ 0.126368] gcov: version magic: 0x41383552 [ 0.128236] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.129082] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.130429] pinctrl core: initialized pinctrl subsystem [ 0.132360] [ 0.132903] ************************************************************* [ 0.135013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.138012] ** ** [ 0.141011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.144014] ** ** [ 0.146010] ** This means that this kernel is built to expose internal ** [ 0.149010] ** IOMMU data structures, which may compromise security on ** [ 0.151010] ** your system. ** [ 0.154013] ** ** [ 0.156010] ** If you see this message and you are not debugging the ** [ 0.159012] ** kernel, report this immediately to your vendor! ** [ 0.161015] ** ** [ 0.163011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166011] ************************************************************* [ 0.169282] NET: Registered protocol family 16 [ 0.171579] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.174062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.177061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.181090] cpuidle: using governor menu [ 0.184009] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.187464] PCI: Using configuration type 1 for base access [ 0.189362] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.199055] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.200024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.202200] cryptd: max_cpu_qlen set to 1000 [ 0.204262] ACPI: Added _OSI(Module Device) [ 0.205014] ACPI: Added _OSI(Processor Device) [ 0.207012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.212019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.227844] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.248966] ACPI: Interpreter enabled [ 0.252068] ACPI: PM: (supports S0 S3 S4 S5) [ 0.256014] ACPI: Using IOAPIC for interrupt routing [ 0.260704] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.270131] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.302124] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.308048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.312020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.316484] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.319000] acpiphp: Slot [2] registered [ 0.320180] acpiphp: Slot [5] registered [ 0.321114] acpiphp: Slot [6] registered [ 0.323138] acpiphp: Slot [7] registered [ 0.324112] acpiphp: Slot [8] registered [ 0.327228] acpiphp: Slot [9] registered [ 0.329251] acpiphp: Slot [10] registered [ 0.333244] acpiphp: Slot [3] registered [ 0.336136] acpiphp: Slot [4] registered [ 0.338194] acpiphp: Slot [11] registered [ 0.342165] acpiphp: Slot [12] registered [ 0.343103] acpiphp: Slot [13] registered [ 0.346130] acpiphp: Slot [14] registered [ 0.349117] acpiphp: Slot [15] registered [ 0.352088] acpiphp: Slot [16] registered [ 0.355132] acpiphp: Slot [17] registered [ 0.358129] acpiphp: Slot [18] registered [ 0.366134] acpiphp: Slot [19] registered [ 0.367099] acpiphp: Slot [20] registered [ 0.373113] acpiphp: Slot [21] registered [ 0.374129] acpiphp: Slot [22] registered [ 0.385045] acpiphp: Slot [23] registered [ 0.387108] acpiphp: Slot [24] registered [ 0.390168] acpiphp: Slot [25] registered [ 0.393683] acpiphp: Slot [26] registered [ 0.396127] acpiphp: Slot [27] registered [ 0.399400] acpiphp: Slot [28] registered [ 0.403676] acpiphp: Slot [29] registered [ 0.406128] acpiphp: Slot [30] registered [ 0.409144] acpiphp: Slot [31] registered [ 0.412077] PCI host bridge to bus 0000:00 [ 0.414022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.420037] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.423052] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.426028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.429024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.432030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.435190] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.438645] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.441485] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.453936] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.459057] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.462018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.465017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.468060] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.471662] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.475013] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.478043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.481992] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.488017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.502024] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.508019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.514623] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.523015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.533025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.573025] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.589589] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.598033] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.612030] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.643024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.663141] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.676022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.687037] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.732032] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.758349] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.768033] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.786030] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.816021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.826688] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.839026] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.852029] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.884025] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.903817] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.920023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.934020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.982028] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.004381] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.006393] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.008456] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.014709] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.021442] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.036037] iommu: Default domain type: Passthrough [ 1.037633] SCSI subsystem initialized [ 1.039131] ACPI: bus type USB registered [ 1.040204] usbcore: registered new interface driver usbfs [ 1.042120] usbcore: registered new interface driver hub [ 1.046114] usbcore: registered new device driver usb [ 1.051228] pps_core: LinuxPPS API ver. 1 registered [ 1.052014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.057083] PTP clock support registered [ 1.061000] EDAC MC: Ver: 3.0.0 [ 1.061000] PCI: Using ACPI for IRQ routing [ 1.061000] NetLabel: Initializing [ 1.061000] NetLabel: domain hash size = 128 [ 1.065021] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.069137] NetLabel: unlabeled traffic allowed by default [ 1.071701] vgaarb: loaded [ 1.078000] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.080015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.093144] clocksource: Switched to clocksource kvm-clock [ 1.338898] VFS: Disk quotas dquot_6.6.0 [ 1.340146] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.345092] *** VALIDATE ramfs *** [ 1.348269] *** VALIDATE hugetlbfs *** [ 1.351106] pnp: PnP ACPI init [ 1.355716] pnp: PnP ACPI: found 6 devices [ 1.401703] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.407526] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.413135] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.415218] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.419877] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.427165] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.429901] NET: Registered protocol family 2 [ 1.435342] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.450032] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.456235] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.465222] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.473632] TCP: Hash tables configured (established 65536 bind 65536) [ 1.483268] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.488927] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.494922] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.499972] NET: Registered protocol family 1 [ 1.502936] RPC: Registered named UNIX socket transport module. [ 1.508639] RPC: Registered udp transport module. [ 1.509854] RPC: Registered tcp transport module. [ 1.512067] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.518213] NET: Registered protocol family 44 [ 1.519498] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.521808] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.524817] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.529375] PCI: CLS 0 bytes, default 64 [ 1.535930] Unpacking initramfs... [ 6.901768] debug: unmapping init [mem 0xffff8fae3cc54000-0xffff8fae3ffbffff] [ 6.932485] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 6.943217] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 6.956620] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 10.059058] Initialise system trusted keyrings [ 10.061294] Key type blacklist registered [ 10.066308] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 10.087617] zbud: loaded [ 10.099454] *** VALIDATE nfs *** [ 10.100526] *** VALIDATE nfs4 *** [ 10.107874] pstore: using deflate compression [ 10.120112] Platform Keyring initialized [ 10.495332] NET: Registered protocol family 38 [ 10.498481] Key type asymmetric registered [ 10.502712] Asymmetric key parser 'x509' registered [ 10.505308] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 10.510373] io scheduler mq-deadline registered [ 10.516881] io scheduler kyber registered [ 10.519595] io scheduler bfq registered [ 10.521618] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 10.531523] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 10.536583] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 10.539115] ACPI: Power Button [PWRF] [ 10.551933] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 10.563948] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 10.592688] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 10.600693] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 10.615710] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 10.641564] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 10.670925] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 10.675302] Non-volatile memory driver v1.3 [ 10.676700] Linux agpgart interface v0.103 [ 10.709265] virtio_blk virtio1: [vda] 134136 512-byte logical blocks (68.7 MB/65.5 MiB) [ 10.713508] vda: detected capacity change from 0 to 68677632 [ 10.736363] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 10.739394] vdb: detected capacity change from 0 to 1073741824 [ 10.773590] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 10.778599] vdc: detected capacity change from 0 to 2621440000 [ 10.821871] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 10.829632] vdd: detected capacity change from 0 to 2621440000 [ 10.872307] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 10.883685] vde: detected capacity change from 0 to 4294967296 [ 10.937849] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 10.947713] vdf: detected capacity change from 0 to 4294967296 [ 10.976472] libphy: Fixed MDIO Bus: probed [ 11.017892] usbcore: registered new interface driver usbserial_generic [ 11.026769] usbserial: USB Serial support registered for generic [ 11.033247] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 11.044783] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 11.054306] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 11.067933] mousedev: PS/2 mouse device common for all mice [ 11.086920] rtc_cmos 00:05: RTC can wake from S4 [ 11.101741] rtc_cmos 00:05: registered as rtc0 [ 11.107446] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 11.119700] intel_pstate: CPU model not supported [ 11.124161] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 11.142292] hid: raw HID events driver (C) Jiri Kosina [ 11.149601] usbcore: registered new interface driver usbhid [ 11.151564] usbhid: USB HID core driver [ 11.159687] drop_monitor: Initializing network drop monitor service [ 11.179769] Initializing XFRM netlink socket [ 11.181928] NET: Registered protocol family 10 [ 11.206162] Segment Routing with IPv6 [ 11.207347] NET: Registered protocol family 17 [ 11.212353] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 11.220045] mpls_gso: MPLS GSO support [ 11.244414] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 11.269622] RAS: Correctable Errors collector initialized. [ 11.275079] AVX version of gcm_enc/dec engaged. [ 11.285193] AES CTR mode by8 optimization enabled [ 11.682445] sched_clock: Marking stable (11682423802, 0)->(13035060622, -1352636820) [ 11.719690] registered taskstats version 1 [ 11.724859] Loading compiled-in X.509 certificates [ 11.730625] zswap: loaded using pool lzo/zbud [ 11.881239] Key type big_key registered [ 11.945863] Key type encrypted registered [ 11.947179] ima: No TPM chip found, activating TPM-bypass! [ 11.948717] ima: Allocated hash algorithm: sha1 [ 11.959343] ima: No architecture policies found [ 11.964762] evm: Initialising EVM extended attributes: [ 11.970561] evm: security.selinux [ 11.979862] evm: security.ima [ 11.984199] evm: security.capability [ 11.991732] evm: HMAC attrs: 0x1 [ 12.002697] rtc_cmos 00:05: setting system clock to 2026-04-10 18:42:54 UTC (1775846574) [ 12.041556] debug: unmapping init [mem 0xffffffffad603000-0xffffffffad7fffff] [ 12.056196] debug: unmapping init [mem 0xffffffffac382000-0xffffffffac658fff] [ 12.097444] Write protecting the kernel read-only data: 28672k [ 12.122695] debug: unmapping init [mem 0xffffffffaaa03000-0xffffffffaabfffff] [ 12.129475] debug: unmapping init [mem 0xffffffffab314000-0xffffffffab3fffff] [ 12.260180] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.295848] systemd[1]: Detected virtualization kvm. [ 12.313141] systemd[1]: Detected architecture x86-64. [ 12.324560] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 12.515791] systemd[1]: No hostname configured. [ 12.525457] systemd[1]: Set hostname to . [ 12.527614] random: systemd: uninitialized urandom read (16 bytes read) [ 12.542984] systemd[1]: Initializing machine ID from random generator. [ 12.740282] random: ln: uninitialized urandom read (6 bytes read) [ 13.221656] random: systemd: uninitialized urandom read (16 bytes read) [ 13.232553] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 13.259876] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 13.278114] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 15.859963] device-mapper: uevent: version 1.0.3 [ 15.864247] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon.[ 18.170098] random: fast init done Starting dracut initqueue hook... [ 18.329424] virtio_net virtio0 ens2: renamed from eth0 [ 18.653624] scsi host0: ata_piix [ 18.699112] scsi host1: ata_piix [ 18.700198] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 18.714057] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 23.514154] random: crng init done [ 23.518621] random: 7 urandom warning(s) missed due to ratelimiting [ 25.954614] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 28.233626] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 32.251321] printk: systemd: 26 output lines suppressed due to ratelimiting [ 33.075225] SELinux: Disabled at runtime. [ 33.243960] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 33.276833] systemd[1]: Detected virtualization kvm. [ 33.280487] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 36.158976] systemd[1]: initrd-switch-root.service: Succeeded. [ 36.178086] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 36.216943] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 36.241592] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 36.248371] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 36.323147] systemd[1]: Starting Journal Service... Starting Journal Service... [ 36.373542] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices.[ 36.983090] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 38.658632] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 39.722880] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 39.835541] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 40.394460] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 40.501938] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit)[ 47.315747] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit)[ 48.693791] NFS: Registering the id_resolver key type [ 48.697796] Key type id_resolver registered [ 48.699897] Key type id_legacy registered [ *] A start job is running for Configur…only root support (13s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Login Service. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg257-server login: [ 89.953817] libcfs: loading out-of-tree module taints kernel. [ 89.972606] Key type ._llcrypt registered [ 89.973840] Key type .llcrypt registered [ 90.034924] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 101.274980] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 102.010143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 102.019859] alg: No test for adler32 (adler32-zlib) [ 103.183285] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 103.634226] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 105.335792] Key type lgssc registered [ 106.298714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 117.950522] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 148.602472] hrtimer: interrupt took 5199269 ns [ 149.681597] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 159.654359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 159.694920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 160.929697] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 160.968326] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 161.064894] Lustre: lustre-MDT0000: new disk, initializing [ 161.166535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 161.187187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 164.507122] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 174.345688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 174.410528] Lustre: 6481:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 174.429604] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 174.432334] Lustre: Skipped 1 previous similar message [ 174.487511] Lustre: lustre-MDT0001: new disk, initializing [ 174.533134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 174.555352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 174.560480] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 177.259802] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 181.072970] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 186.814263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 186.956969] Lustre: lustre-OST0000: new disk, initializing [ 186.958827] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 186.988160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 190.286344] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 191.491772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 191.498506] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 191.563061] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 200.766727] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 200.860333] Lustre: lustre-OST0001: new disk, initializing [ 200.863334] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 200.909402] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 205.086414] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 208.915563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 208.924227] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 208.976762] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 213.616396] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 219.315373] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 229.308513] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing check_logdir /tmp/testlogs/ [ 232.191296] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing yml_node [ 235.099820] Lustre: DEBUG MARKER: Client: 2.17.52.1 [ 236.724962] Lustre: DEBUG MARKER: MDS: 2.17.52.1 [ 238.631552] Lustre: DEBUG MARKER: OSS: 2.17.52.1 [ 239.789984] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Apr 10 14:46:40 EDT 2026 [ 251.780059] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 252.911602] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 266.719702] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 266.722985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 266.729247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 270.310973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 270.313704] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 270.317778] Lustre: Skipped 2 previous similar messages [ 270.341693] Lustre: Skipped 2 previous similar messages [ 272.395482] Lustre: server umount lustre-MDT0000 complete [ 274.432453] LustreError: 6474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775846836 with bad export cookie 18302683247327334597 [ 274.435991] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 274.438222] LustreError: 6474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 275.424731] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 275.425301] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 275.431757] Lustre: Skipped 1 previous similar message [ 275.439607] Lustre: Skipped 2 previous similar messages [ 279.457841] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 279.463136] Lustre: Skipped 1 previous similar message [ 280.684969] Lustre: server umount lustre-MDT0001 complete [ 283.063666] Lustre: server umount lustre-OST0000 complete [ 285.554314] Lustre: server umount lustre-OST0001 complete [ 289.281221] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 294.147275] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 315.602218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 315.780171] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 315.807455] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 315.863245] Lustre: lustre-MDT0000: new disk, initializing [ 315.918417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 315.930186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 318.234466] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 324.412598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 324.496262] Lustre: 14944:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 324.551923] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 324.555779] Lustre: Skipped 1 previous similar message [ 324.640374] Lustre: lustre-MDT0001: new disk, initializing [ 324.715705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 324.743485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 324.753518] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 327.413194] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 332.149722] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 334.441507] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 338.110551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 339.539162] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 339.602122] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 341.285508] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 346.499492] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 349.516447] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 349.697799] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 353.279589] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 353.496889] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 355.299403] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 355.306318] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 355.307483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 355.312538] Lustre: Skipped 1 previous similar message [ 365.540183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 365.548901] Lustre: Skipped 4 previous similar messages [ 369.119121] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 369.232134] Lustre: server umount lustre-OST0000 complete [ 375.779651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 375.779748] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 375.790971] Lustre: Skipped 1 previous similar message [ 377.773168] Lustre: server umount lustre-MDT0000 complete [ 380.080877] LustreError: 14935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775846942 with bad export cookie 18302683247327336130 [ 380.082463] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 380.087852] LustreError: 14935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 380.325824] Lustre: server umount lustre-MDT0001 complete [ 383.597953] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 14:49:04 (1775846944) [ 394.191161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 394.458850] LustreError: 19415:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 394.464154] LustreError: 19415:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 394.498888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 394.502827] Lustre: Skipped 1 previous similar message [ 396.544126] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 399.847157] LustreError: 19416:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 401.305513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 403.581841] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 406.781781] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 408.645335] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 409.948492] Lustre: Failing over lustre-MDT0000 [ 410.099563] Lustre: server umount lustre-MDT0000 complete [ 410.979375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 410.981950] LustreError: 19412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 410.994129] Lustre: Skipped 1 previous similar message [ 411.029263] LustreError: 19412:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 416.226730] LustreError: 20677:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 416.241366] LustreError: 20677:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 421.344512] LustreError: 19412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 421.352461] LustreError: 19412:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 425.543851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 425.631714] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 426.848451] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 426.852871] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 426.862471] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 427.894737] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 432.262670] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 437.217194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 437.218735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 437.226608] Lustre: Skipped 1 previous similar message [ 437.234661] Lustre: Skipped 3 previous similar messages [ 441.188080] Lustre: server umount lustre-MDT0000 complete [ 442.336662] LustreError: 19416:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 442.344437] LustreError: 19416:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 443.277121] LustreError: 19395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775847005 with bad export cookie 18302683247327337544 [ 443.279363] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 443.287709] LustreError: 19395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 443.448121] Lustre: server umount lustre-MDT0001 complete [ 447.314912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 447.585309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 447.591416] Lustre: Skipped 2 previous similar messages [ 449.590757] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 454.147572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 456.757250] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 460.422232] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 462.562675] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 464.005524] Lustre: Failing over lustre-MDT0000 [ 464.169267] Lustre: server umount lustre-MDT0000 complete [ 464.872708] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 464.890547] LustreError: 22407:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 464.903045] LustreError: 22407:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 480.039853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 480.129350] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 483.269275] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 485.345798] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 485.347986] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 485.355842] Lustre: Skipped 1 previous similar message [ 485.362777] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 488.282658] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 495.590148] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 495.591914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 495.592800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 495.604161] Lustre: Skipped 3 previous similar messages [ 505.311148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 505.438704] Lustre: server umount lustre-MDT0000 complete [ 507.457535] LustreError: 22388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775847069 with bad export cookie 18302683247327338482 [ 507.460900] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 507.464665] LustreError: 22388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 507.674627] Lustre: server umount lustre-MDT0001 complete [ 512.759474] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 14:51:13 (1775847073) [ 522.017520] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 526.589757] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 549.184758] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 555.009641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 555.140054] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 555.160609] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 555.162978] Lustre: Skipped 1 previous similar message [ 555.211616] Lustre: lustre-MDT0000: new disk, initializing [ 555.214221] Lustre: Skipped 1 previous similar message [ 555.296472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 555.300535] Lustre: Skipped 2 previous similar messages [ 555.317695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 555.322765] Lustre: Skipped 1 previous similar message [ 557.555991] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 564.567472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 564.633979] Lustre: 28381:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 564.791444] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 567.040230] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 571.436487] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 573.351466] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 576.974807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 578.661995] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 578.710587] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 578.723889] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 579.945860] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 583.776830] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 587.206034] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 599.521861] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 599.524508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 599.525785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 599.529798] Lustre: Skipped 1 previous similar message [ 599.531629] Lustre: Skipped 5 previous similar messages [ 601.406868] Lustre: server umount lustre-OST0000 complete [ 609.316200] Lustre: server umount lustre-MDT0000 complete [ 609.759940] LustreError: 28391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 609.766267] LustreError: 28391:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 611.125158] LustreError: 28371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775847173 with bad export cookie 18302683247327338888 [ 611.127994] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 611.129800] LustreError: 28371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 611.329966] Lustre: server umount lustre-MDT0001 complete [ 616.835066] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 618.309646] Key type lgssc unregistered [ 618.523860] LNet: 31943:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 618.530196] LNetError: 31943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 618.542470] LNet: Removed LNI 192.168.202.157@tcp [ 619.049152] Key type .llcrypt unregistered [ 619.051398] Key type ._llcrypt unregistered [ 632.599114] Key type ._llcrypt registered [ 632.600881] Key type .llcrypt registered [ 632.649927] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 633.022053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 633.030287] alg: No test for adler32 (adler32-zlib) [ 633.949858] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 634.040433] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 635.631472] Key type lgssc registered [ 636.166386] Lustre: Echo OBD driver; http://www.lustre.org/ [ 640.381834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 640.394336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 641.657118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 643.709832] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 648.621891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 648.830791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 650.699526] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 653.758577] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 655.597721] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 658.716082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 658.954263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 660.016954] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 661.844330] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 664.760510] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 677.309520] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 14:53:58 (1775847238) [ 678.405107] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 679.212280] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 682.016938] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 14:54:03 (1775847243) [ 685.800503] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 14:54:06 (1775847246) [ 686.256103] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 690.533688] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 690.536800] Lustre: Skipped 2 previous similar messages [ 691.353724] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 14:54:12 (1775847252) [ 695.954394] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 695.958137] Lustre: Skipped 4 previous similar messages [ 703.939705] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 14:54:24 (1775847264) [ 705.302537] Lustre: *** cfs_fail_loc=131b, val=2*** [ 707.065367] Lustre: *** cfs_fail_loc=131b, val=2*** [ 711.319456] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 14:54:32 (1775847272) [ 712.181203] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 712.184189] Lustre: Skipped 9 previous similar messages [ 718.435341] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 14:54:39 (1775847279) [ 726.497613] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 726.502231] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 726.516780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 728.546730] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 728.553607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 730.634777] Lustre: server umount lustre-OST0000 complete [ 733.664893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 733.667450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 733.671068] Lustre: Skipped 1 previous similar message [ 733.676304] Lustre: Skipped 2 previous similar messages [ 738.662789] Lustre: server umount lustre-MDT0000 complete [ 738.786040] LustreError: 33314:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 738.793958] LustreError: 33314:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 740.437978] LustreError: 33297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775847302 with bad export cookie 3305770507910682469 [ 740.438492] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 740.444583] LustreError: 33297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 740.640261] Lustre: server umount lustre-MDT0001 complete [ 746.286334] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 747.591473] Key type lgssc unregistered [ 747.759427] LNet: 40792:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 747.765035] LNetError: 40792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 747.778233] LNet: Removed LNI 192.168.202.157@tcp [ 748.182065] Key type .llcrypt unregistered [ 748.184140] Key type ._llcrypt unregistered [ 755.624167] Key type ._llcrypt registered [ 755.625643] Key type .llcrypt registered [ 755.666329] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 763.586727] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 764.068075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 764.152810] alg: No test for adler32 (adler32-zlib) [ 765.063177] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 765.207659] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 766.823207] Key type lgssc registered [ 767.356870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 784.964876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 784.985080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 786.126428] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 786.162784] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 786.218717] Lustre: lustre-MDT0000: new disk, initializing [ 786.296215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 786.310992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 788.499443] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 794.457186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 794.498222] Lustre: 43883:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 794.522952] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 794.525765] Lustre: Skipped 1 previous similar message [ 794.565696] Lustre: lustre-MDT0001: new disk, initializing [ 794.611624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 794.628079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 794.632832] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 796.547239] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 800.995719] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 802.965076] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 806.955977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 807.119095] Lustre: lustre-OST0000: new disk, initializing [ 807.122228] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 807.172744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 809.941236] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 814.188863] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 815.635239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 815.644573] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 815.709438] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 816.277494] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 816.362975] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 818.247132] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 818.354393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 820.710795] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 820.711832] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 820.720969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 820.724509] Lustre: Skipped 1 previous similar message [ 825.320158] Lustre: server umount lustre-OST0000 complete [ 830.432084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 830.435451] LustreError: Skipped 1 previous similar message [ 830.439054] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 830.450600] Lustre: Skipped 1 previous similar message [ 830.454965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 833.808118] Lustre: server umount lustre-MDT0000 complete [ 835.687215] LustreError: 43877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775847398 with bad export cookie 2713932824035801406 [ 835.691404] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 835.692941] LustreError: 43877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 835.877751] Lustre: server umount lustre-MDT0001 complete [ 840.852664] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 14:56:41 (1775847401) [ 844.192032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 844.493294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 846.283912] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 849.888597] LustreError: 47931:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 849.897934] LustreError: 47931:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 850.171634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 850.329763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 852.121957] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 854.846310] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 856.667794] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 860.114310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 860.306687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 863.581315] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 866.421355] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 878.745758] LustreError: 50144:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 879.176561] Lustre: Modifying parameter general.timeout=1 in log params [ 880.449304] Lustre: Modifying parameter general.timeout=4 in log params [ 880.451376] Lustre: Skipped 2 previous similar messages [ 882.512903] Lustre: Modifying parameter general.timeout=9 in log params [ 882.515060] Lustre: Skipped 4 previous similar messages [ 886.554917] Lustre: Modifying parameter general.timeout=19 in log params [ 886.557062] Lustre: Skipped 9 previous similar messages [ 894.866838] Lustre: Modifying parameter general.timeout=41 in log params [ 894.869060] Lustre: Skipped 21 previous similar messages [ 911.257883] Lustre: Modifying parameter general.timeout=79 in log params [ 911.263690] Lustre: Skipped 37 previous similar messages [ 939.528471] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 14:58:20 (1775847500) [ 940.783635] Lustre: 57057:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 942.717115] Lustre: 57253:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 942.722723] Lustre: 57253:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 944.221178] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 in log params [ 944.227885] Lustre: Skipped 56 previous similar messages [ 944.814794] Lustre: 57469:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 944.821310] Lustre: 57469:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 959.216995] Lustre: 58922:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 959.221930] Lustre: 58922:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 959.233475] LustreError: 58922:0:(mgs_handler.c:1164:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 961.213030] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 14:58:42 (1775847522) [ 968.159862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 968.162787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 968.167397] Lustre: Skipped 1 previous similar message [ 968.169472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 968.172019] Lustre: Skipped 1 previous similar message [ 970.720702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 970.722149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 970.732790] Lustre: Skipped 1 previous similar message [ 972.999578] Lustre: server umount lustre-MDT0000 complete [ 974.748469] LustreError: 47912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775847537 with bad export cookie 2713932824035802274 [ 974.749314] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 974.756927] LustreError: 47912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 974.906762] Lustre: server umount lustre-MDT0001 complete [ 986.931145] Lustre: server umount lustre-OST0000 complete [ 1000.807195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1000.957899] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1001.109861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1003.330951] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1007.787476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1007.847896] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1007.885530] Lustre: 61450:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1010.031276] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1012.871233] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1014.742189] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1018.034194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1018.128463] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1020.999603] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1023.658551] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1024.441896] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1024.446026] Lustre: Skipped 17 previous similar messages [ 1024.462376] Lustre: 63670:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1029.090046] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1029.096604] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1029.097265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1029.102951] Lustre: Skipped 3 previous similar messages [ 1032.802253] Lustre: server umount lustre-OST0000 complete [ 1038.816533] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1038.821864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1038.826052] Lustre: Skipped 1 previous similar message [ 1038.828895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1049.055125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1049.138436] Lustre: server umount lustre-MDT0000 complete [ 1050.801369] LustreError: 61440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775847613 with bad export cookie 2713932824035804829 [ 1050.803884] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1050.806737] LustreError: 61440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1050.954956] Lustre: server umount lustre-MDT0001 complete [ 1055.374987] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 1056.923933] Key type lgssc unregistered [ 1057.065469] LNet: 64861:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1057.071411] LNetError: 64861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1058.085648] LNet: Removed LNI 192.168.202.157@tcp [ 1058.470609] Key type .llcrypt unregistered [ 1058.472267] Key type ._llcrypt unregistered [ 1070.992891] Key type ._llcrypt registered [ 1070.993838] Key type .llcrypt registered [ 1071.050356] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 1071.564039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1071.593136] alg: No test for adler32 (adler32-zlib) [ 1072.489935] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 1072.605459] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 1074.231135] Key type lgssc registered [ 1074.778451] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1080.315707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1080.340955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1081.865811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1084.927837] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1089.302220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1089.580886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1091.952662] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1095.003218] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1097.098405] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1101.505019] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1101.837440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1105.698833] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1108.761957] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1118.551037] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1124.320752] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1124.324305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1124.335021] Lustre: Skipped 2 previous similar messages [ 1124.340083] Lustre: Skipped 1 previous similar message [ 1124.764627] Lustre: server umount lustre-MDT0000 complete [ 1126.874165] LustreError: 68593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775847689 with bad export cookie 7035998720691959637 [ 1126.882164] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1126.882951] LustreError: 68593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1129.308608] Lustre: server umount lustre-MDT0001 complete [ 1148.588080] Lustre: server umount lustre-OST0000 complete [ 1177.929296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1178.323280] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1178.369061] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1178.610900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1182.819954] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1190.698261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1190.863938] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1190.919256] Lustre: 70781:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1191.423299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1195.070551] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1200.926498] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1204.265916] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1210.420886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1210.607652] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1210.966469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1216.219351] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1220.805892] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1223.157946] Lustre: 73013:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1225.667499] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1230.308428] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1230.320653] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1230.347677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1230.372077] Lustre: Skipped 1 previous similar message [ 1230.836491] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1235.954190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1235.970872] Lustre: Skipped 2 previous similar messages [ 1236.619648] Lustre: server umount lustre-OST0000 complete [ 1241.061711] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1241.063401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1241.069611] Lustre: Skipped 2 previous similar messages [ 1241.086645] Lustre: Skipped 1 previous similar message [ 1246.177052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1246.185616] Lustre: Skipped 1 previous similar message [ 1246.930663] Lustre: server umount lustre-MDT0000 complete [ 1250.764493] LustreError: 70771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775847813 with bad export cookie 7035998720691960435 [ 1250.788641] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1251.094341] Lustre: server umount lustre-MDT0001 complete [ 1260.376453] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 1263.033992] Key type lgssc unregistered [ 1263.273055] LNet: 74208:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1263.278358] LNetError: 74208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1263.285938] LNet: Removed LNI 192.168.202.157@tcp [ 1264.210520] Key type .llcrypt unregistered [ 1264.216567] Key type ._llcrypt unregistered [ 1285.156200] Key type ._llcrypt registered [ 1285.158118] Key type .llcrypt registered [ 1285.235274] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 1286.624508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1286.669576] alg: No test for adler32 (adler32-zlib) [ 1287.672318] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 1287.877898] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 1289.503184] Key type lgssc registered [ 1290.523231] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1297.219761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1297.230759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1298.726514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1302.499729] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1309.158264] LustreError: 75604:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1309.178607] LustreError: 75604:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1309.929711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1310.419257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1313.876418] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1319.053667] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1322.341612] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1328.327242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1328.843842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1334.318911] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1338.891391] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1348.519914] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1349.252983] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1350.850576] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1350.856402] Lustre: Skipped 1 previous similar message [ 1353.064322] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1353.067920] Lustre: Skipped 2 previous similar messages [ 1357.438414] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1357.441643] Lustre: Skipped 5 previous similar messages [ 1365.860172] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1365.863420] Lustre: Skipped 10 previous similar messages [ 1382.260783] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1382.268904] Lustre: Skipped 18 previous similar messages [ 1414.628494] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1414.636794] Lustre: Skipped 38 previous similar messages [ 1438.532962] Lustre: 83361:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1440.989507] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 15:06:41 (1775848001) [ 1442.435656] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1444.705127] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 15:06:44 (1775848004) [ 1452.003674] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1452.014130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1452.026910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1452.040658] Lustre: Skipped 1 previous similar message [ 1454.279825] Lustre: server umount lustre-OST0000 complete [ 1458.672103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1458.687548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1458.694515] Lustre: Skipped 1 previous similar message [ 1458.697995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1462.240955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1462.243346] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1463.963857] Lustre: server umount lustre-MDT0000 complete [ 1467.366272] LustreError: 79826:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1467.388478] LustreError: 79826:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1467.509295] LustreError: 75583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775848030 with bad export cookie 6375177801399532873 [ 1467.518770] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1467.537579] LustreError: 75583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1467.833234] Lustre: server umount lustre-MDT0001 complete [ 1477.730765] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 1480.199347] Key type lgssc unregistered [ 1480.442398] LNet: 84592:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1480.452551] LNetError: 84592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1480.472642] LNet: Removed LNI 192.168.202.157@tcp [ 1481.221417] Key type .llcrypt unregistered [ 1481.223832] Key type ._llcrypt unregistered [ 1485.194979] Key type ._llcrypt registered [ 1485.197592] Key type .llcrypt registered [ 1485.280755] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1488.577636] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules [ 1489.400605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1489.427125] alg: No test for adler32 (adler32-zlib) [ 1490.623320] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 1490.764560] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 1492.415229] Key type lgssc registered [ 1493.358367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1494.317718] LustreError: 85761:0:(lu_object.c:1477:lu_context_key_register()) cfs_race id 60d sleeping [ 1507.749166] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 1516.075462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1516.406543] LustreError: 85761:0:(lu_object.c:1477:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1516.435634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1517.951593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1520.988410] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1526.825642] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 15:08:07 (1775848087) [ 1528.300898] LustreError: 86765:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1528.319507] LustreError: 86765:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1530.482674] LustreError: 86797:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1530.489570] LustreError: 86797:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1530.494720] LustreError: 86797:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 12, retries 0, failed: rc = -5 [ 1530.759080] Lustre: server umount lustre-MDT0000 complete [ 1539.600407] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 1541.407389] Key type lgssc unregistered [ 1541.627845] LNet: 88254:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1541.636945] LNetError: 88254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1541.652747] LNet: Removed LNI 192.168.202.157@tcp [ 1542.159716] Key type .llcrypt unregistered [ 1542.162049] Key type ._llcrypt unregistered [ 1555.127811] Key type ._llcrypt registered [ 1555.128950] Key type .llcrypt registered [ 1555.182747] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 1555.846681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1555.916849] alg: No test for adler32 (adler32-zlib) [ 1556.906961] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 1557.063292] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 1558.679195] Key type lgssc registered [ 1559.517878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1565.344404] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1565.365098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1566.797244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1570.531211] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1576.936077] LustreError: 89433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1576.951342] LustreError: 89433:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1577.023289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1577.460226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1582.272891] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1591.465749] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1595.414558] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1605.069129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1605.780153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1614.673768] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1622.746798] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1666.797399] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 15:10:27 (1775848227) [ 1668.376589] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1669.943579] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 15:10:30 (1775848230) [ 1674.725719] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1674.734481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1674.748694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1678.818859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1678.828177] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1678.851679] Lustre: Skipped 1 previous similar message [ 1680.809523] Lustre: server umount lustre-MDT0000 complete [ 1683.778174] LustreError: 89414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775848246 with bad export cookie 6847891306080756278 [ 1683.790293] LustreError: 89414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1683.795875] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1683.936190] LustreError: 91057:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1683.937974] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1683.952933] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1683.956720] Lustre: Skipped 2 previous similar messages [ 1684.106310] Lustre: server umount lustre-MDT0001 complete [ 1693.484506] Lustre: server umount lustre-OST0000 complete [ 1702.664614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1703.044847] LustreError: 93374:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1703.203306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1706.154360] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1708.512654] LustreError: 93375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1712.574530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1712.609856] LustreError: 93374:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1713.093680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1716.637245] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1721.012727] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1723.586801] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1733.689920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1733.807136] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1733.811623] LustreError: 93362:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1733.828350] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1733.833754] LustreError: 95228:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -98 [ 1733.837464] LustreError: 95228:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 1733.840615] LustreError: 95228:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 1733.892236] Lustre: server umount lustre-OST0000 complete [ 1733.893820] LustreError: 95228:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1739.661524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1739.762876] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1739.766145] LustreError: 93362:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1739.770164] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1739.774210] LustreError: 95689:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -98 [ 1739.777665] LustreError: 95689:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 1739.781144] LustreError: 95689:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 1739.831920] Lustre: server umount lustre-OST0000 complete [ 1739.834160] LustreError: 95689:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1745.671344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1745.785435] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1745.792587] Lustre: Found index 0 for lustre-OST0000, updating log [ 1745.796878] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1745.847435] Lustre: lustre-OST0000: new disk, initializing [ 1745.850440] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1746.026346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1749.925833] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1752.665281] Lustre: Failing over lustre-OST0000 [ 1752.803246] Lustre: server umount lustre-OST0000 complete [ 1757.665167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1757.666594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1757.677117] Lustre: Skipped 1 previous similar message [ 1762.784212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1762.787993] Lustre: Skipped 2 previous similar messages [ 1765.906675] Lustre: server umount lustre-MDT0000 complete [ 1767.906232] LustreError: 93375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1767.920765] LustreError: 93375:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1768.389509] LustreError: 96223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775848330 with bad export cookie 6847891306080759022 [ 1768.392629] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1768.398352] LustreError: 96223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1768.613084] Lustre: server umount lustre-MDT0001 complete [ 1773.383825] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 15:12:14 (1775848334) [ 1782.393428] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 1793.002656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1793.321607] LustreError: 98537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1793.509083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1799.983450] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1813.479310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1820.470491] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1824.607385] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1834.243637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1835.654532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1835.657632] Lustre: Skipped 1 previous similar message [ 1835.679521] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1837.683861] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1837.754811] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1837.794698] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 1837.830067] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 1848.362669] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1862.831851] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1862.921957] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1862.990534] Lustre: lustre-OST0001: new disk, initializing [ 1862.994044] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1869.735575] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1870.417160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1870.424842] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1870.589441] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1877.607681] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1887.944500] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1897.903083] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.57@tcp (stopping) [ 1900.000391] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1900.006673] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1903.517464] Lustre: server umount lustre-MDT0001 complete [ 1906.670046] LustreError: 98533:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1906.699867] LustreError: 98533:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 1913.999042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1914.159308] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1914.173246] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1914.261502] Lustre: 98527:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1914.933900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1914.938230] Lustre: Skipped 1 previous similar message [ 1915.847410] LustreError: 102379:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1915.856796] LustreError: 102379:0:(tgt_mount.c:666:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1915.863251] LustreError: 98529:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1915.922655] LustreError: 102384:0:(obd_config.c:2101:class_config_llog_handler()) MGC192.168.202.157@tcp: cfg command failed: rc = -17 [ 1915.928443] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1915.928443] [ 1918.923925] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1919.987601] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1928.878077] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 15:14:49 (1775848489) [ 1945.791474] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 1964.834699] Lustre: 104382:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1993.189424] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1997.206785] Lustre: 105518:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2088.928243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2088.940582] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2088.953086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2088.954221] Lustre: Skipped 5 previous similar messages [ 2089.005001] Lustre: Skipped 8 previous similar messages [ 2093.604845] Lustre: server umount lustre-MDT0000 complete [ 2094.057071] LustreError: 105686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2094.081956] LustreError: 105686:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2097.780658] LustreError: 99645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775848660 with bad export cookie 6847891306080759953 [ 2097.791910] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2098.531035] Lustre: server umount lustre-MDT0001 complete [ 2114.330953] Lustre: server umount lustre-OST0000 complete [ 2115.104136] Lustre: 88748:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775848661/real 1775848661] req@ffff8faea5d5c000 x1862111717586944/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775848677 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2115.145162] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2117.603951] Lustre: 88745:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775848664/real 1775848664] req@ffff8fad82e3d500 x1862111717587200/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775848680 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2118.993288] Lustre: server umount lustre-OST0001 complete [ 2125.217050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2135.564448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2145.427247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2155.979677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2177.416079] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 2188.163740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2188.182471] Lustre: lustre-MDT0000: reset Object Index mappings [ 2188.996661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2193.769895] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2202.752454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2202.787071] Lustre: lustre-MDT0001: reset Object Index mappings [ 2207.473640] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2210.578571] Lustre: 110576:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2218.444313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2226.835519] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2228.540439] LustreError: 110929:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2228.572132] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 2228.582884] LustreError: 110929:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2234.890298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 2237.570682] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2239.810563] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2239.834646] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2245.513621] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2254.884518] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2259.027874] Lustre: 112427:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2311.274031] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 15:21:11 (1775848871) [ 2340.835090] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2340.838466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2340.860133] Lustre: Skipped 3 previous similar messages [ 2340.865232] Lustre: Skipped 2 previous similar messages [ 2344.803619] Lustre: server umount lustre-MDT0000 complete [ 2348.409408] LustreError: 109441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775848910 with bad export cookie 6847891306080807084 [ 2348.415863] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2348.423448] LustreError: 109441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2355.042906] Lustre: server umount lustre-MDT0001 complete [ 2373.090684] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2373.209727] Lustre: server umount lustre-OST0000 complete [ 2384.541522] Lustre: server umount lustre-OST0001 complete [ 2392.316480] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 2403.855886] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 2440.629078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2440.859923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2440.881495] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2440.941337] Lustre: lustre-MDT0000: new disk, initializing [ 2441.067762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2445.298073] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2452.297661] Lustre: server umount lustre-MDT0000 complete [ 2461.802112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2462.067532] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2462.082257] Lustre: 116651:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2462.095616] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2462.293161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2462.300422] Lustre: Skipped 4 previous similar messages [ 2467.521579] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2472.988951] Lustre: server umount lustre-MDT0000 complete [ 2481.326545] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 15:24:01 (1775849041) [ 2482.700245] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2484.598746] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 15:24:04 (1775849044) [ 2500.579968] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 2512.449494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2517.262516] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2527.644558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2527.893325] Lustre: 118669:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2527.990149] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2528.005458] Lustre: Skipped 1 previous similar message [ 2528.161919] Lustre: lustre-MDT0001: new disk, initializing [ 2528.444268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2528.466744] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2534.641841] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2540.454830] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2546.979784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2547.337402] Lustre: lustre-OST0000: new disk, initializing [ 2547.353590] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2548.592732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2548.603735] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2548.797513] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2553.419636] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2565.820769] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2565.997807] Lustre: lustre-OST0001: new disk, initializing [ 2566.002089] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2567.596843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2567.614747] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2567.716489] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2572.163585] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2582.131228] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2585.827437] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2950.629623] Lustre: 127459:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2950.634440] Lustre: 127459:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2950.637529] Lustre: 127459:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2950.641360] Lustre: 127459:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2950.645678] Lustre: 127459:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2950.651042] Lustre: 127459:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3214.942723] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 15:36:14 (1775849774) [ 3229.579398] Lustre: server umount lustre-MDT0000 complete [ 3229.664092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3229.669983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3229.678657] Lustre: Skipped 2 previous similar messages [ 3229.682061] LustreError: 120207:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3229.693125] LustreError: 120207:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 3233.179612] LustreError: 120213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775849795 with bad export cookie 6847891306080873892 [ 3233.185806] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3233.190896] LustreError: 120213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3233.257913] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3233.274823] Lustre: Skipped 6 previous similar messages [ 3239.689537] Lustre: server umount lustre-MDT0001 complete [ 3257.311342] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3257.531812] Lustre: server umount lustre-OST0000 complete [ 3267.974671] Lustre: server umount lustre-OST0001 complete [ 3275.904856] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 3287.326514] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 3325.077966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3325.254507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3325.314950] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3325.377255] Lustre: lustre-MDT0000: new disk, initializing [ 3325.497672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3325.502804] Lustre: Skipped 4 previous similar messages [ 3325.520704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3329.951922] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 3340.992909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3341.086398] Lustre: 130473:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3341.136992] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3341.140882] Lustre: Skipped 1 previous similar message [ 3341.212924] Lustre: lustre-MDT0001: new disk, initializing [ 3341.385992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3341.401968] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3345.939841] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 3354.058714] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3357.853400] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3363.801823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3363.971851] Lustre: lustre-OST0000: new disk, initializing [ 3363.979097] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3365.315835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3365.329835] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3365.420316] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3369.236860] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 3375.611715] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3383.800915] Lustre: lustre-MDD0000: changelog on [ 3385.626337] Lustre: lustre-MDD0001: changelog on [ 5169.152663] Lustre: lustre-MDD0001: changelog off [ 5171.239755] Lustre: 130481:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5172.066577] Lustre: lustre-MDD0000: changelog off [ 5182.945592] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5182.970753] Lustre: Skipped 4 previous similar messages [ 5182.971739] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5182.972662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5182.972667] Lustre: Skipped 2 previous similar messages [ 5185.651542] Lustre: server umount lustre-OST0000 complete [ 5189.601487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5189.614588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5189.650136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5189.659697] Lustre: Skipped 1 previous similar message [ 5193.188056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5193.197751] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5195.691071] Lustre: server umount lustre-MDT0000 complete [ 5198.304280] LustreError: 132222:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5198.325162] LustreError: 132222:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5199.713820] LustreError: 130466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775851762 with bad export cookie 6847891306081692465 [ 5199.722369] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5199.727880] LustreError: 130466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5200.157379] Lustre: server umount lustre-MDT0001 complete [ 5211.451208] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 5214.346944] Key type lgssc unregistered [ 5214.685286] LNet: 141215:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5214.698578] LNetError: 141215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5214.721172] LNet: Removed LNI 192.168.202.157@tcp [ 5215.385169] Key type .llcrypt unregistered [ 5215.390500] Key type ._llcrypt unregistered [ 5238.480261] Key type ._llcrypt registered [ 5238.483570] Key type .llcrypt registered [ 5238.577934] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 5256.158588] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 5257.773155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5257.891176] alg: No test for adler32 (adler32-zlib) [ 5259.348410] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 5259.806540] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 5261.639329] Key type lgssc registered [ 5263.398444] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5307.864496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5307.907877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5309.225539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5309.259548] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5309.357893] Lustre: lustre-MDT0000: new disk, initializing [ 5309.466674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5309.491866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5314.348472] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 5326.059913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5326.251620] Lustre: 144384:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5326.320186] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5326.330292] Lustre: Skipped 1 previous similar message [ 5326.508670] Lustre: lustre-MDT0001: new disk, initializing [ 5326.725059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5326.761602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5326.766700] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5331.913951] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 5340.879234] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5346.273392] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5355.204174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5355.714519] Lustre: lustre-OST0000: new disk, initializing [ 5355.719200] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5355.898859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5363.288645] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 5363.784798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5363.794368] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5363.967580] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5371.922789] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5377.583872] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_module obdecho/obdecho [ 5379.906270] LustreError: 146999:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5379.919990] LustreError: 146999:0:(obd_config.c:843:class_setup()) setup ec failed (-22) [ 5381.736627] LustreError: 147096:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 5393.390094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5393.403973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5393.424075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5394.922634] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5394.935644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5394.945367] Lustre: Skipped 1 previous similar message [ 5399.050548] Lustre: server umount lustre-MDT0000 complete [ 5403.368502] LustreError: 144377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775851965 with bad export cookie 12423087995673912393 [ 5403.372081] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5403.383162] LustreError: 144377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5403.818850] Lustre: server umount lustre-MDT0001 complete [ 5418.427768] Lustre: server umount lustre-OST0000 complete [ 5443.063820] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 5454.548955] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 5497.433306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5497.734716] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5497.795411] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5497.900490] Lustre: lustre-MDT0000: new disk, initializing [ 5498.013721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5498.060878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5505.064792] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 5518.649910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5518.824958] Lustre: 150779:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5518.890631] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5518.901154] Lustre: Skipped 1 previous similar message [ 5519.074737] Lustre: lustre-MDT0001: new disk, initializing [ 5519.257666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5519.360391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5519.373682] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5524.920461] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 5533.626812] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5538.095878] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5546.231245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5546.565774] Lustre: lustre-OST0000: new disk, initializing [ 5546.575477] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5546.693517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5553.622023] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 5553.745943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5553.758302] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5554.005578] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5562.928695] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5568.415699] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5568.717351] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5573.993675] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5574.207550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5579.751919] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5579.753682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5579.757718] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5579.763069] Lustre: Skipped 2 previous similar messages [ 5579.794110] Lustre: Skipped 2 previous similar messages [ 5582.832235] Lustre: server umount lustre-OST0000 complete [ 5589.988565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5589.995934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5590.008847] Lustre: Skipped 1 previous similar message [ 5590.022224] Lustre: Skipped 1 previous similar message [ 5595.107820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5595.119716] Lustre: Skipped 1 previous similar message [ 5601.247207] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5601.591421] Lustre: server umount lustre-MDT0000 complete [ 5605.344741] LustreError: 150786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5605.362941] LustreError: 150786:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5605.407139] LustreError: 150771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775852167 with bad export cookie 12423087995673913345 [ 5605.412984] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5605.416357] LustreError: 150771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5605.673200] Lustre: server umount lustre-MDT0001 complete [ 5609.169301] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 16:16:09 (1775852169) [ 5610.840452] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 5613.053806] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 16:16:13 (1775852173) [ 5621.635861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5622.296424] LustreError: 154792:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5622.435524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5626.050064] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 5627.874068] LustreError: 154793:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5631.980582] LustreError: 154792:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5635.237669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5640.947090] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 5646.341703] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5650.560850] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5658.067263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5658.434785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5658.445541] Lustre: Skipped 1 previous similar message [ 5665.244360] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 5673.431101] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5684.660986] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.57@tcp (stopping) [ 5684.673467] Lustre: Skipped 2 previous similar messages [ 5686.156700] Lustre: server umount lustre-MDT0000 complete [ 5686.757950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5686.766058] LustreError: Skipped 1 previous similar message [ 5686.777275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5686.797713] LustreError: 154792:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5690.317287] LustreError: 154772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775852252 with bad export cookie 12423087995673914234 [ 5690.329134] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5690.332150] LustreError: 154772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5694.863965] LustreError: 154787:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5694.886704] LustreError: 154787:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5696.960073] Lustre: server umount lustre-MDT0001 complete [ 5704.800442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5720.553833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5720.897956] LustreError: 158297:0:(llog.c:1646:llog_backup()) MGC192.168.202.157@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5720.911744] Lustre: 158297:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.202.157@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5721.129129] LustreError: 158302:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5721.155655] LustreError: 158302:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5756.907558] LustreError: 158304:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5756.930763] LustreError: 158304:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 26 previous similar messages [ 5781.471616] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 5781.478738] Lustre: 158297:0:(mgc_request.c:1917:mgc_process_log()) MGC192.168.202.157@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 5821.835795] LustreError: 158302:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5821.865021] LustreError: 158302:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 44 previous similar messages [ 5822.456771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5827.081539] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 5827.557189] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5837.314754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5851.110870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 5866.764600] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 5866.994772] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5873.832746] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5876.216806] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5881.107435] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5882.944771] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5905.208159] Lustre: server umount lustre-MDT0000 complete [ 5907.938276] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5907.960123] Lustre: Skipped 4 previous similar messages [ 5909.192522] LustreError: 154774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775852471 with bad export cookie 12423087995673920723 [ 5909.205351] LustreError: 154774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5909.215232] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5909.535795] Lustre: server umount lustre-MDT0001 complete [ 5924.502625] Lustre: server umount lustre-OST0000 complete [ 5949.903147] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 5965.264716] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 6007.977266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6008.346907] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6008.401417] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6008.491786] Lustre: lustre-MDT0000: new disk, initializing [ 6008.592372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6008.596962] Lustre: Skipped 1 previous similar message [ 6008.616603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6013.590430] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6024.447519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6024.588318] Lustre: 163172:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6024.828282] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6029.884338] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6038.605161] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6043.326258] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6050.775825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6051.097410] Lustre: lustre-OST0000: new disk, initializing [ 6051.101412] Lustre: Skipped 1 previous similar message [ 6051.109403] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6051.121359] Lustre: Skipped 2 previous similar messages [ 6057.872676] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6059.616260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6059.635538] Lustre: Skipped 1 previous similar message [ 6059.643255] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6059.755047] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6066.388841] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6072.135549] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6072.372150] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6077.800706] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6078.006819] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6080.483318] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6080.488197] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6080.505853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6080.505860] Lustre: Skipped 3 previous similar messages [ 6080.533311] Lustre: Skipped 2 previous similar messages [ 6086.570093] Lustre: server umount lustre-OST0000 complete [ 6091.232505] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6091.239336] LustreError: Skipped 1 previous similar message [ 6097.602603] Lustre: server umount lustre-MDT0000 complete [ 6100.976555] LustreError: 163179:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6101.002685] LustreError: 163179:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [ 6102.552895] LustreError: 164912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775852665 with bad export cookie 12423087995673937523 [ 6102.559655] LustreError: 164912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6102.559778] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6103.133755] Lustre: server umount lustre-MDT0001 complete [ 6114.481565] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 16:24:34 (1775852674) [ 6121.517831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6127.275529] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6135.549867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6141.303309] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6147.775482] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6152.580824] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6161.490521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6169.727424] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6176.337057] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6193.394578] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 16:25:52 (1775852752) [ 6197.223720] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6197.238705] Lustre: Skipped 2 previous similar messages [ 6197.266432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6197.269256] Lustre: Skipped 6 previous similar messages [ 6203.073190] Lustre: server umount lustre-OST0000 complete [ 6213.899950] Lustre: server umount lustre-MDT0000 complete [ 6217.791068] LustreError: 167162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775852780 with bad export cookie 12423087995673938405 [ 6217.797307] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6217.803386] LustreError: 167162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6218.109048] Lustre: server umount lustre-MDT0001 complete [ 6229.007722] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 6232.167520] Key type lgssc unregistered [ 6232.520055] LNet: 170559:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6232.533184] LNetError: 170559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6232.566350] LNet: Removed LNI 192.168.202.157@tcp [ 6233.660828] Key type .llcrypt unregistered [ 6233.663360] Key type ._llcrypt unregistered [ 6254.055191] Key type ._llcrypt registered [ 6254.058873] Key type .llcrypt registered [ 6254.211106] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 6255.825315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6255.856640] alg: No test for adler32 (adler32-zlib) [ 6257.013753] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 6257.240615] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 6258.967206] Key type lgssc registered [ 6260.513250] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6268.833614] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6268.871607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6284.575356] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6289.695242] LustreError: 171786:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.157@tcp: failed processing log, type 4: rc = -110 [ 6294.815429] LustreError: 171786:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6294.830801] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6294.836039] LustreError: MGC192.168.202.157@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6294.846191] LustreError: 171786:0:(tgt_mount.c:1819:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6294.853289] LustreError: 171786:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -22 [ 6294.859365] LustreError: 171786:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 6295.061970] Lustre: server umount lustre-OST0000 complete [ 6295.075041] LustreError: 171786:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6306.040065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6307.821191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6313.750550] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6318.056358] LustreError: 172285:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6318.096252] LustreError: 172285:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6323.170585] LustreError: 172286:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6325.089729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6325.622702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6331.097573] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6337.319592] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6341.201040] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6348.762198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6349.172271] LustreError: 173876:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6349.193259] Lustre: 173876:0:(mgc_request_server.c:743:mgc_llog_local_copy()) MGC192.168.202.157@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6349.402361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6355.859841] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6362.279685] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6366.689066] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6366.714205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6366.742644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6369.248501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6369.254399] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6371.095971] Lustre: server umount lustre-MDT0000 complete [ 6374.372474] LustreError: 172286:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6374.409485] LustreError: 172286:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6375.791889] LustreError: 172267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775852938 with bad export cookie 13192711029696914714 [ 6375.805139] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6375.810392] LustreError: 172267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6376.945836] Lustre: server umount lustre-MDT0001 complete [ 6393.311903] Lustre: server umount lustre-OST0000 complete [ 6404.079552] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 6416.816334] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 6454.315021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6454.542221] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6454.577449] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6454.715460] Lustre: lustre-MDT0000: new disk, initializing [ 6454.861155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6454.877479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6459.460213] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6471.117986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6471.272953] Lustre: 177261:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6471.350478] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6471.353645] Lustre: Skipped 1 previous similar message [ 6471.414924] Lustre: lustre-MDT0001: new disk, initializing [ 6471.518862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6471.554067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6471.566279] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6475.581078] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6483.548488] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6487.160381] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6494.032064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6494.244704] Lustre: lustre-OST0000: new disk, initializing [ 6494.248121] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6494.310749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6501.266532] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6503.009338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6503.021799] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6503.116072] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6508.666778] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6513.483825] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6513.809346] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6518.163428] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6518.403361] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6523.872921] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6523.878154] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6523.897504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6523.926971] Lustre: Skipped 2 previous similar messages [ 6523.942535] Lustre: Skipped 3 previous similar messages [ 6526.258495] Lustre: server umount lustre-OST0000 complete [ 6532.065783] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6532.079843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6537.065078] Lustre: server umount lustre-MDT0000 complete [ 6538.217026] LustreError: 178449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6538.245043] LustreError: 178449:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6541.538823] LustreError: 177252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775853104 with bad export cookie 13192711029696915435 [ 6541.555701] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6541.564447] LustreError: 177252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 6541.877780] Lustre: server umount lustre-MDT0001 complete [ 6553.693390] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 16:31:53 (1775853113) [ 6573.763980] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 6576.744769] Key type lgssc unregistered [ 6577.009887] LNet: 181782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6577.018392] LNetError: 181782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6577.039214] LNet: Removed LNI 192.168.202.157@tcp [ 6577.772919] Key type .llcrypt unregistered [ 6577.777269] Key type ._llcrypt unregistered [ 6596.535347] Key type ._llcrypt registered [ 6596.543400] Key type .llcrypt registered [ 6596.740612] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 6598.107282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6598.315062] alg: No test for adler32 (adler32-zlib) [ 6599.434882] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 6599.680351] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 6601.321975] Key type lgssc registered [ 6602.639533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6610.903913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6610.930231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6612.662253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6617.863482] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6623.200760] LustreError: 182986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6623.235797] LustreError: 182986:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6628.329321] LustreError: 182985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6630.545800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6631.071488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6636.281890] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6642.822556] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6646.956226] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6655.119036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6655.371768] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6655.376713] LustreError: 184623:0:(llog.c:1687:llog_backup()) MGC192.168.202.157@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6655.397879] Lustre: 184623:0:(mgc_request_server.c:740:mgc_llog_local_copy()) MGC192.168.202.157@tcp: can't backup local config lustre-OST0000: rc = -28 [ 6655.576214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6661.460132] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6666.658222] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6671.844229] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6671.853794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6671.874675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6673.387067] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6673.388180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6673.427334] Lustre: Skipped 2 previous similar messages [ 6675.809935] Lustre: server umount lustre-MDT0000 complete [ 6678.501115] LustreError: 183696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6678.532925] LustreError: 183696:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6680.168084] LustreError: 185273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775853242 with bad export cookie 5673127691746015158 [ 6680.175684] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6680.187796] LustreError: 185273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6680.466872] Lustre: server umount lustre-MDT0001 complete [ 6693.832267] Lustre: server umount lustre-OST0000 complete [ 6704.337147] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 6717.658637] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 6755.838335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6756.089929] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6756.121936] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6756.208371] Lustre: lustre-MDT0000: new disk, initializing [ 6756.283494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6756.300739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6761.922860] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6776.280653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6776.487382] Lustre: 188006:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6776.680838] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6776.685494] Lustre: Skipped 1 previous similar message [ 6776.697499] LustreError: 188772:0:(llog.c:1687:llog_backup()) MGC192.168.202.157@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 6776.708693] LustreError: 188772:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 6776.717740] Lustre: 188772:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.202.157@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 6776.743885] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6776.751519] Lustre: Skipped 1 previous similar message [ 6776.904966] Lustre: lustre-MDT0001: new disk, initializing [ 6777.193752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6777.266409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6777.280210] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6782.528410] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6791.260440] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6795.409890] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6803.050816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6803.437957] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6803.441888] Lustre: Skipped 1 previous similar message [ 6803.445956] LustreError: 189726:0:(llog.c:1687:llog_backup()) MGC192.168.202.157@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6803.457311] LustreError: 189726:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 6803.468783] Lustre: 189726:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.202.157@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 6803.487590] Lustre: 189726:0:(mgc_request_server.c:763:mgc_llog_local_copy()) Skipped 1 previous similar message [ 6803.521833] Lustre: lustre-OST0000: new disk, initializing [ 6803.531422] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6803.681788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6810.462324] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6811.218026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6811.236824] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6811.395471] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6818.136963] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6822.794410] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6823.026175] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6827.918354] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6828.163300] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6832.099892] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6832.101250] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6832.125320] Lustre: Skipped 1 previous similar message [ 6832.141279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6832.183394] Lustre: Skipped 1 previous similar message [ 6837.217691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6837.233033] Lustre: Skipped 1 previous similar message [ 6842.353943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6842.373231] Lustre: Skipped 1 previous similar message [ 6844.383158] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6844.677560] Lustre: server umount lustre-OST0000 complete [ 6848.994394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6849.008083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6852.578335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6852.589687] Lustre: Skipped 2 previous similar messages [ 6863.327152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6863.704800] Lustre: server umount lustre-MDT0000 complete [ 6867.414721] LustreError: 187998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775853429 with bad export cookie 5673127691746015879 [ 6867.417814] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6867.428767] LustreError: 187998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6867.820686] Lustre: server umount lustre-MDT0001 complete [ 6878.230361] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 16:37:17 (1775853437) [ 6902.986723] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 6916.382774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6917.301064] LustreError: 192735:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6917.419938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6922.689842] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6922.721695] LustreError: 192736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6931.831042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6931.938967] LustreError: 192735:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6931.962218] LustreError: 192735:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6936.633244] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6940.572805] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6949.262382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6949.840280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6949.843576] Lustre: Skipped 1 previous similar message [ 6957.554633] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6967.110413] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6967.252790] Lustre: lustre-OST0001: new disk, initializing [ 6967.258863] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6974.547654] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6976.102140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6976.113819] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6976.236718] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6986.785693] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6997.840878] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7004.945281] Lustre: DEBUG MARKER: ADD OST3 [ 7011.426238] loop: module loaded [ 7011.489904] loop0: detected capacity change from 0 to 204800000 [ 7011.720919] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7011.751040] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7011.770183] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7019.591286] loop0: detected capacity change from 0 to 204800000 [ 7019.805632] Lustre: DEBUG MARKER: STOP OST3 [ 7028.467676] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7028.786297] Lustre: lustre-OST0003: new disk, initializing [ 7028.791799] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7028.939297] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7028.948293] Lustre: Skipped 1 previous similar message [ 7029.970288] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7030.125923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7030.135693] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7030.192410] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7032.038592] LustreError: 197111:0:(osp_precreate.c:1280:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7032.485727] Lustre: Failing over lustre-OST0003 [ 7032.682534] Lustre: server umount lustre-OST0003 complete [ 7035.373694] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7035.388229] LustreError: 194187:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7035.398350] Lustre: Skipped 2 previous similar messages [ 7035.408026] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7037.118128] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7045.326172] Lustre: DEBUG MARKER: START OST3 again [ 7052.953981] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7053.341756] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7054.873932] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7055.040324] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7055.043823] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7059.883578] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7082.232055] Lustre: Failing over lustre-OST0003 [ 7082.475580] Lustre: server umount lustre-OST0003 complete [ 7084.020046] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7084.023200] LustreError: 194187:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7084.038394] Lustre: Skipped 1 previous similar message [ 7084.061802] LustreError: 194187:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 7104.484909] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7104.492303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7104.510061] Lustre: Skipped 2 previous similar messages [ 7104.531082] Lustre: Skipped 7 previous similar messages [ 7109.642166] Lustre: server umount lustre-MDT0000 complete [ 7113.695187] LustreError: 195749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775853676 with bad export cookie 5673127691746016761 [ 7113.696789] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7113.710620] LustreError: 195749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7128.033542] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7128.523518] Lustre: server umount lustre-MDT0001 complete [ 7132.782618] Lustre: server umount lustre-OST0000 complete [ 7137.039371] Lustre: server umount lustre-OST0001 complete [ 7143.737413] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 7153.052081] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 7198.758101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7198.976530] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7199.007373] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7199.068835] Lustre: lustre-MDT0000: new disk, initializing [ 7199.127043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7199.130166] Lustre: Skipped 1 previous similar message [ 7199.149185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7203.611285] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7214.204939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7214.318211] Lustre: 202383:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7214.556869] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7218.914170] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7228.152078] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7232.424732] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7240.444317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7240.814392] Lustre: lustre-OST0000: new disk, initializing [ 7240.817011] Lustre: Skipped 1 previous similar message [ 7240.822311] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7240.832399] Lustre: Skipped 2 previous similar messages [ 7242.237754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7242.251806] Lustre: Skipped 1 previous similar message [ 7242.261783] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7242.358248] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7247.272419] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7257.010615] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7263.512567] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7263.801941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7268.747784] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7268.967602] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7272.928867] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7272.937191] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7272.959811] Lustre: Skipped 4 previous similar messages [ 7272.971238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7272.989502] Lustre: Skipped 7 previous similar messages [ 7285.216053] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7285.595629] Lustre: server umount lustre-OST0000 complete [ 7291.360639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7296.730510] Lustre: server umount lustre-MDT0000 complete [ 7298.530675] LustreError: 203245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7298.558218] LustreError: 203245:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [ 7300.651373] LustreError: 203159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775853863 with bad export cookie 5673127691746019057 [ 7300.654178] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7300.664501] LustreError: 203159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7300.892463] Lustre: server umount lustre-MDT0001 complete [ 7305.993920] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 16:44:24 (1775853864) [ 7323.621839] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 7336.250521] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 7380.105477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7380.453566] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7380.517119] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7380.701334] Lustre: lustre-MDT0000: new disk, initializing [ 7380.975609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7385.309931] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7397.870031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7398.106519] Lustre: 208559:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7398.505321] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7404.059185] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7413.541859] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7418.612593] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7427.655420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7429.612653] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7436.200446] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7445.325213] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7450.596295] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7450.917770] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7456.027873] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7456.235962] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7460.345893] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7460.353129] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7460.372990] LustreError: Skipped 1 previous similar message [ 7460.383700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7460.391230] Lustre: Skipped 3 previous similar messages [ 7460.419238] Lustre: Skipped 8 previous similar messages [ 7465.473119] LustreError: 210375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7465.770162] Lustre: server umount lustre-OST0000 complete [ 7476.971850] Lustre: server umount lustre-MDT0000 complete [ 7481.041757] LustreError: 209337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775854043 with bad export cookie 5673127691746019939 [ 7481.044160] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7481.052088] LustreError: 209337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7481.392095] Lustre: server umount lustre-MDT0001 complete [ 7491.373602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7492.174391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7492.184652] Lustre: Skipped 5 previous similar messages [ 7498.255212] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7508.591665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7514.396440] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7519.861744] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7524.286279] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7532.276315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7539.251699] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7545.730902] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7557.601379] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7557.613568] LustreError: Skipped 1 previous similar message [ 7561.973354] Lustre: server umount lustre-OST0000 complete [ 7571.945315] Lustre: server umount lustre-MDT0000 complete [ 7575.474413] LustreError: 212355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775854137 with bad export cookie 5673127691746020821 [ 7575.495257] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7575.823332] Lustre: server umount lustre-MDT0001 complete [ 7684.899780] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 7687.651815] Key type lgssc unregistered [ 7687.886720] LNet: 215683:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7687.890766] LNetError: 215683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7687.902170] LNet: Removed LNI 192.168.202.157@tcp [ 7688.661157] Key type .llcrypt unregistered [ 7688.663479] Key type ._llcrypt unregistered [ 7714.642301] Key type ._llcrypt registered [ 7714.644616] Key type .llcrypt registered [ 7714.741494] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 7730.350692] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 7731.365061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7731.489049] alg: No test for adler32 (adler32-zlib) [ 7732.585547] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 7732.801371] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 7734.527257] Key type lgssc registered [ 7736.211624] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7775.012527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7775.051063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7776.523752] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7776.547836] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7776.647670] Lustre: lustre-MDT0000: new disk, initializing [ 7776.759794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7776.779288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7781.744228] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7793.486583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7793.612552] Lustre: 219003:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7793.685811] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7793.691615] Lustre: Skipped 1 previous similar message [ 7793.798165] Lustre: lustre-MDT0001: new disk, initializing [ 7793.962975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7794.025857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7794.042558] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7799.143191] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7807.151530] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7810.794309] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7817.777575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7818.085462] Lustre: lustre-OST0000: new disk, initializing [ 7818.093128] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7818.216054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7824.989109] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7828.133313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7828.153980] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7828.312209] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7833.416736] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7837.701881] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7837.958370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7842.256160] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7842.478481] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7848.928504] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7848.946944] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7848.960395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7850.676676] Lustre: server umount lustre-OST0000 complete [ 7855.584588] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7855.588284] LustreError: Skipped 1 previous similar message [ 7855.600641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7855.619095] Lustre: Skipped 1 previous similar message [ 7855.625174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7855.627745] Lustre: Skipped 1 previous similar message [ 7859.187647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7859.199982] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7861.294261] Lustre: server umount lustre-MDT0000 complete [ 7865.793408] LustreError: 218996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775854428 with bad export cookie 7296126609243005815 [ 7865.801033] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7865.811307] LustreError: 218996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7866.337441] Lustre: server umount lustre-MDT0001 complete [ 7887.539840] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 7897.857430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7898.408317] LustreError: 223513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7898.441689] LustreError: 223513:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7898.472511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7903.481240] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7903.715651] LustreError: 223514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7907.819649] LustreError: 223513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7912.932582] LustreError: 223514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7913.118446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7913.609252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7918.933822] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7922.222313] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7930.282429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7930.682907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7937.657498] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7946.240526] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7946.429267] Lustre: lustre-OST0001: new disk, initializing [ 7946.436752] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7953.274600] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7956.051173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7956.059955] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7956.187867] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7965.090398] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7973.324282] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7993.262856] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 16:55:53 (1775854553) [ 7997.419754] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7997.429712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7997.443598] Lustre: Skipped 2 previous similar messages [ 7997.452058] Lustre: Skipped 2 previous similar messages [ 8001.814505] Lustre: server umount lustre-MDT0000 complete [ 8002.541688] LustreError: 223508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8002.569164] LustreError: 223508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8006.084705] LustreError: 225772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775854568 with bad export cookie 7296126609243006697 [ 8006.089853] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8006.097324] LustreError: 225772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8006.464574] Lustre: server umount lustre-MDT0001 complete [ 8020.370774] Lustre: server umount lustre-OST0000 complete [ 8024.042389] Lustre: 217231:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775854570/real 1775854570] req@ffff8fad82297b80 x1862118193347072/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775854586 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8024.092129] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8024.109017] Lustre: Skipped 1 previous similar message [ 8025.220442] Lustre: server umount lustre-OST0001 complete [ 8032.295208] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 8042.367490] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 8080.877494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8081.371771] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8081.405823] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8081.538771] Lustre: lustre-MDT0000: new disk, initializing [ 8081.651327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8081.655532] Lustre: Skipped 1 previous similar message [ 8081.693765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8087.251401] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8100.602923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8100.901393] Lustre: 229697:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8100.979135] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8100.981799] Lustre: Skipped 1 previous similar message [ 8101.053388] Lustre: lustre-MDT0001: new disk, initializing [ 8101.207283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8101.228388] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8107.444555] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8115.773751] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8119.891248] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8127.906978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8128.213090] Lustre: lustre-OST0000: new disk, initializing [ 8128.219343] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8128.331251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8128.337764] Lustre: Skipped 1 previous similar message [ 8130.120981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8130.132335] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8130.287724] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8135.422617] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8144.127472] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8149.586354] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8149.943362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8155.450425] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8155.703393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8161.250192] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8161.266983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8161.273979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8161.284253] Lustre: Skipped 1 previous similar message [ 8161.302282] Lustre: Skipped 2 previous similar messages [ 8171.493418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8171.518028] Lustre: Skipped 4 previous similar messages [ 8171.999145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8172.271319] Lustre: server umount lustre-OST0000 complete [ 8182.965809] Lustre: server umount lustre-MDT0000 complete [ 8186.857343] LustreError: 231444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8186.895497] LustreError: 231444:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8189.068780] LustreError: 229688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775854751 with bad export cookie 7296126609243008839 [ 8189.070779] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8189.090563] LustreError: 229688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8189.786186] Lustre: server umount lustre-MDT0001 complete [ 8201.038965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8201.569989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8206.742476] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8206.821510] LustreError: 233509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8206.862620] LustreError: 233509:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8216.283912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8220.905832] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8227.763454] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8231.488568] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8239.536072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8246.507829] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8254.202888] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8405.988053] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8405.994149] LustreError: Skipped 1 previous similar message [ 8405.998850] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8406.002656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8406.017721] Lustre: Skipped 3 previous similar messages [ 8406.039727] Lustre: Skipped 5 previous similar messages [ 8407.559718] Lustre: server umount lustre-OST0000 complete [ 8411.104377] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8411.107989] LustreError: Skipped 1 previous similar message [ 8417.571976] Lustre: server umount lustre-MDT0000 complete [ 8419.297016] LustreError: 233505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8419.312561] LustreError: 233505:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8421.703414] LustreError: 233489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775854984 with bad export cookie 7296126609243009721 [ 8421.716547] LustreError: 233489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8421.718975] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8422.487946] Lustre: server umount lustre-MDT0001 complete [ 8526.406478] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 8530.313300] Key type lgssc unregistered [ 8530.700767] LNet: 236776:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8530.713411] LNetError: 236776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8530.730736] LNet: Removed LNI 192.168.202.157@tcp [ 8531.887325] Key type .llcrypt unregistered [ 8531.897234] Key type ._llcrypt unregistered [ 8560.078577] Key type ._llcrypt registered [ 8560.081518] Key type .llcrypt registered [ 8560.193927] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 8577.314284] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 8578.800579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8578.907394] alg: No test for adler32 (adler32-zlib) [ 8579.987361] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 8580.202802] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 8581.879249] Key type lgssc registered [ 8583.401391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8636.625948] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 8653.437253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8653.485507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8654.916984] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8655.024528] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8655.150953] Lustre: lustre-MDT0000: new disk, initializing [ 8655.250794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8655.269788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8660.624615] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8674.548530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8674.665304] Lustre: 240803:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8674.849256] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8674.858987] Lustre: Skipped 1 previous similar message [ 8675.018304] Lustre: lustre-MDT0001: new disk, initializing [ 8675.195721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8675.254747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8675.270462] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8679.991754] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8685.374720] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8694.238775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8694.536100] Lustre: lustre-OST0000: new disk, initializing [ 8694.545606] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8694.693532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8695.783872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8695.795929] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8695.985618] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8701.701738] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8712.975746] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8713.187488] Lustre: lustre-OST0001: new disk, initializing [ 8713.193267] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8713.335605] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8719.392892] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8720.540473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8720.546191] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8720.662152] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8732.313649] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8740.222841] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8761.840613] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8761.850239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8761.863268] Lustre: Skipped 1 previous similar message [ 8761.882726] Lustre: Skipped 3 previous similar messages [ 8766.329253] Lustre: server umount lustre-MDT0000 complete [ 8770.777170] LustreError: 242401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775855333 with bad export cookie 10528045901510419141 [ 8770.787902] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8770.792279] LustreError: 242401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8771.292288] Lustre: server umount lustre-MDT0001 complete [ 8784.731424] Lustre: server umount lustre-OST0000 complete [ 8787.938871] Lustre: 238327:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775855334/real 1775855334] req@ffff8faeb6794000 x1862119081461248/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775855350 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8787.961448] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8787.971538] Lustre: Skipped 2 previous similar messages [ 8788.885506] Lustre: server umount lustre-OST0001 complete [ 8803.689088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8814.455070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8823.045559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8834.709612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8854.697413] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 8866.841976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8866.871463] Lustre: lustre-MDT0000: reset Object Index mappings [ 8867.546617] LustreError: 248232:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8867.572424] LustreError: 248232:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 8867.673394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8872.761657] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8872.934946] LustreError: 248233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8878.056614] LustreError: 248232:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8882.439932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8882.458976] Lustre: lustre-MDT0001: reset Object Index mappings [ 8882.905800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8888.287821] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8891.937140] Lustre: 249344:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8899.537336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8899.938546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8907.038107] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8908.139786] LustreError: 249696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8913.253465] LustreError: 249698:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8913.274527] LustreError: 249698:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8913.278805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8916.553880] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8918.529764] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 8924.495729] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 8934.181384] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8938.313612] Lustre: 251190:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8952.803789] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8952.813310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8952.823276] Lustre: Skipped 3 previous similar messages [ 8952.828027] Lustre: Skipped 2 previous similar messages [ 8957.925400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8957.943157] Lustre: Skipped 4 previous similar messages [ 8958.372209] Lustre: server umount lustre-MDT0000 complete [ 8962.410265] LustreError: 248213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775855524 with bad export cookie 10528045901510425931 [ 8962.417330] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8962.432442] LustreError: 248213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8962.781009] Lustre: server umount lustre-MDT0001 complete [ 8977.569409] Lustre: server umount lustre-OST0000 complete [ 8979.427212] Lustre: 238326:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775855525/real 1775855525] req@ffff8fae88d81180 x1862119081517184/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775855541 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8979.461373] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8981.520985] Lustre: server umount lustre-OST0001 complete [ 9002.310523] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 17:12:42 (1775855562) [ 9019.114429] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 9030.301413] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 9073.368213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9073.641430] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9073.666836] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9073.739860] Lustre: lustre-MDT0000: new disk, initializing [ 9073.819508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9073.824694] Lustre: Skipped 1 previous similar message [ 9073.841123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9079.319692] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 9091.590970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9091.719919] Lustre: 255428:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9091.777842] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9091.788040] Lustre: Skipped 1 previous similar message [ 9091.885576] Lustre: lustre-MDT0001: new disk, initializing [ 9092.096163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9092.109387] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9097.064265] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 9104.774309] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9108.473403] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9115.458196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9115.729158] Lustre: lustre-OST0000: new disk, initializing [ 9115.733435] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9117.391726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9117.403440] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9117.616175] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9122.926549] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 9130.974824] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9135.829366] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9136.007731] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9140.468955] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9140.702790] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9143.268434] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9143.270445] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9143.280878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9143.316191] Lustre: Skipped 1 previous similar message [ 9148.392761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9148.405867] Lustre: Skipped 1 previous similar message [ 9148.898952] Lustre: server umount lustre-OST0000 complete [ 9153.506200] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9153.513630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9153.537613] Lustre: Skipped 1 previous similar message [ 9158.624820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9158.635713] Lustre: Skipped 4 previous similar messages [ 9159.686427] Lustre: server umount lustre-MDT0000 complete [ 9163.697852] LustreError: 255419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775855726 with bad export cookie 10528045901510428024 [ 9163.706852] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9163.714682] LustreError: 255419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9163.956904] Lustre: server umount lustre-MDT0001 complete [ 9186.753567] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 9201.641397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9202.091556] LustreError: 259939:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9202.150851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9202.157493] Lustre: Skipped 2 previous similar messages [ 9206.992254] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 9215.385041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9220.182728] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 9223.221756] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9229.873313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9237.349513] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 9246.080277] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9246.360867] Lustre: lustre-OST0001: new disk, initializing [ 9246.367677] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9247.506139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9247.525839] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9247.649489] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9253.233139] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 9263.431458] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9266.859707] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9277.966888] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9278.151811] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9287.648583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9287.651739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9287.666150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9291.537252] Lustre: server umount lustre-MDT0000 complete [ 9293.800604] LustreError: 259934:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9293.827152] LustreError: 259934:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9295.785597] LustreError: 261416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775855858 with bad export cookie 10528045901510428906 [ 9295.796521] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9295.806925] LustreError: 261416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9296.296499] Lustre: server umount lustre-MDT0001 complete [ 9311.049706] Lustre: server umount lustre-OST0000 complete [ 9314.978494] Lustre: 238328:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775855861/real 1775855861] req@ffff8fad8d5a9180 x1862119081629568/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775855877 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9315.261391] Lustre: server umount lustre-OST0001 complete [ 9332.936187] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 9344.494050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9345.088339] LustreError: 265224:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9345.193345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9345.196247] Lustre: Skipped 3 previous similar messages [ 9349.602433] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 9360.472367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9367.642627] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 9372.201943] Lustre: 266331:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9381.504798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9389.758795] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 9390.230336] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9402.410812] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9404.502287] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9409.941670] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 9418.411264] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9422.763799] Lustre: 268185:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9433.399314] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9433.579096] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9445.881962] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 17:20:05 (1775856005) [ 9463.268833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9463.286295] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9463.297800] Lustre: Skipped 4 previous similar messages [ 9463.317702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9463.323492] Lustre: Skipped 4 previous similar messages [ 9468.831780] Lustre: server umount lustre-MDT0000 complete [ 9469.410562] LustreError: 265225:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9469.435071] LustreError: 265225:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 9472.972735] LustreError: 265949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775856035 with bad export cookie 10528045901510435668 [ 9472.983710] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9472.990754] LustreError: 265949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9487.327223] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9487.619546] Lustre: server umount lustre-MDT0001 complete [ 9498.357815] Lustre: server umount lustre-OST0000 complete [ 9502.714805] Lustre: server umount lustre-OST0001 complete [ 9509.448467] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 9520.798728] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 9578.971376] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 9592.193485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9592.691624] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9592.729743] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9592.840595] Lustre: lustre-MDT0000: new disk, initializing [ 9593.020571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9598.046394] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 9610.047221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9610.191636] Lustre: 272380:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9610.352501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9610.356785] Lustre: Skipped 4 previous similar messages [ 9610.409781] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9615.043455] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 9629.440942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9631.100750] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9636.837660] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 9649.927761] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9651.859543] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9651.871049] Lustre: Skipped 1 previous similar message [ 9651.987556] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9658.789397] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 9671.967888] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9676.437220] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9676.440263] Lustre: Skipped 1 previous similar message [ 9699.080756] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 17:24:18 (1775856258) [ 9702.880241] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9702.886887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9702.902603] Lustre: Skipped 5 previous similar messages [ 9702.911980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9702.925997] Lustre: Skipped 10 previous similar messages [ 9703.885716] Lustre: server umount lustre-MDT0000 complete [ 9707.953298] LustreError: 273977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775856270 with bad export cookie 10528045901510443123 [ 9707.981696] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9707.986589] LustreError: 273977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9708.457693] Lustre: server umount lustre-MDT0001 complete [ 9723.781831] Lustre: server umount lustre-OST0000 complete [ 9724.704460] Lustre: 238327:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775856271/real 1775856271] req@ffff8faeae939180 x1862119081796992/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775856287 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9728.279905] Lustre: server umount lustre-OST0001 complete [ 9749.555257] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 9763.206138] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 9804.291813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9804.511333] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9804.534747] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9804.536977] Lustre: Skipped 4 previous similar messages [ 9804.711467] Lustre: lustre-MDT0000: new disk, initializing [ 9804.713887] Lustre: Skipped 3 previous similar messages [ 9804.853100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9804.866147] Lustre: Skipped 3 previous similar messages [ 9811.008093] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 9822.519455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9822.724106] Lustre: 279421:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9823.067290] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9828.070715] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 9838.676751] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9843.534888] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9850.935512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9853.410152] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9858.299663] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 9867.069985] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9869.664681] Lustre: Mounted lustre-client [ 9870.523233] LustreError: 281920:0:(lov_obd.c:786:lov_cleanup()) lustre-clilov-ffff8fad8b91d000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 9870.610155] Lustre: Unmounted lustre-client [ 9884.128936] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9884.130595] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9884.143802] LustreError: Skipped 1 previous similar message [ 9884.165944] Lustre: Skipped 4 previous similar messages [ 9884.172979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9884.185354] Lustre: Skipped 5 previous similar messages [ 9894.367157] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9894.591051] Lustre: server umount lustre-OST0000 complete [ 9906.658601] Lustre: server umount lustre-MDT0000 complete [ 9910.752654] LustreError: 279429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9910.774299] LustreError: 279429:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 9911.049087] LustreError: 281174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775856473 with bad export cookie 10528045901510444649 [ 9911.068582] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9911.472215] Lustre: server umount lustre-MDT0001 complete [ 9925.392318] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 9926.883102] LNet: 1 local NIs in recovery (showing 1): 192.168.202.157@tcp [ 9928.823410] Key type lgssc unregistered [ 9929.275346] LNet: 283203:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9929.299966] LNetError: 283203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9999.993669] LNet: Removed LNI 192.168.202.157@tcp [10001.095591] Key type .llcrypt unregistered [10001.100206] Key type ._llcrypt unregistered [10022.922242] Key type ._llcrypt registered [10022.925096] Key type .llcrypt registered [10023.104184] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [10043.771584] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [10045.908207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10045.996276] alg: No test for adler32 (adler32-zlib) [10047.159204] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [10047.500430] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [10049.287467] Key type lgssc registered [10050.970723] Lustre: Echo OBD driver; http://www.lustre.org/ [10087.790843] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 17:30:47 (1775856647) [10095.422341] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10095.452670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10096.886306] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10096.930399] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10097.030390] Lustre: lustre-MDT0000: new disk, initializing [10097.241794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10097.278928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10102.913253] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [10115.589802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10115.722572] Lustre: 286387:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10115.776851] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10115.781248] Lustre: Skipped 1 previous similar message [10115.841365] Lustre: lustre-MDT0001: new disk, initializing [10115.935325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10115.984834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10116.007683] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10121.237603] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [10130.300863] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10135.034964] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10143.220675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10143.684584] Lustre: lustre-OST0000: new disk, initializing [10143.691684] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10143.848300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10152.058514] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [10154.220322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10154.237310] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10154.381524] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10161.572513] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10174.952232] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10174.954634] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10174.964242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10174.978213] Lustre: Skipped 1 previous similar message [10174.995117] Lustre: Skipped 1 previous similar message [10177.954564] Lustre: server umount lustre-OST0000 complete [10182.625527] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10182.638997] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10182.656908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10185.186468] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10185.190303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10185.223022] Lustre: Skipped 1 previous similar message [10188.400047] Lustre: server umount lustre-MDT0000 complete [10192.755376] LustreError: 286379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775856755 with bad export cookie 14902023247161388871 [10192.757538] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10192.766855] LustreError: 286379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10193.138399] Lustre: server umount lustre-MDT0001 complete [10205.342717] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [10208.640321] Key type lgssc unregistered [10208.916570] LNet: 289760:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10208.922454] LNetError: 289760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10208.937216] LNet: Removed LNI 192.168.202.157@tcp [10210.071737] Key type .llcrypt unregistered [10210.074305] Key type ._llcrypt unregistered [10239.655244] Key type ._llcrypt registered [10239.657516] Key type .llcrypt registered [10239.780743] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [10241.159987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10241.205996] alg: No test for adler32 (adler32-zlib) [10242.740573] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [10243.072675] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [10244.711246] Key type lgssc registered [10246.139556] Lustre: Echo OBD driver; http://www.lustre.org/ [10254.762169] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10254.793546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10256.486349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10261.204403] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [10265.633916] LustreError: 291160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10265.678902] LustreError: 291160:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10270.701984] LustreError: 291161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10270.828753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10271.262959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10276.549585] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [10282.909842] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10287.239539] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10295.314852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10295.746440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10303.991376] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [10311.129868] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10320.408947] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10322.227385] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10327.529750] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10327.544088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10327.582197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10329.547336] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.57@tcp (stopping) [10331.622148] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10331.623861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10331.643849] Lustre: Skipped 1 previous similar message [10331.661975] Lustre: Skipped 2 previous similar messages [10334.609211] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.57@tcp (stopping) [10339.728685] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.57@tcp (stopping) [10339.760898] Lustre: Skipped 3 previous similar messages [10341.345721] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10341.555241] Lustre: server umount lustre-MDT0000 complete [10341.863424] LustreError: 291161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10341.897289] LustreError: 291161:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10344.877666] LustreError: 291155:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10345.718392] LustreError: 291142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775856908 with bad export cookie 7096697787072415398 [10345.720417] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10345.733581] LustreError: 291142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10346.986698] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10349.960350] LustreError: 293647:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10349.978554] LustreError: 293647:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10349.984864] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.57@tcp (stopping) [10349.996046] Lustre: Skipped 1 previous similar message [10360.208990] LustreError: 292089:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10360.220205] LustreError: 292089:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10360.287340] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10360.695857] Lustre: server umount lustre-MDT0001 complete [10364.990626] Lustre: server umount lustre-OST0000 complete [10373.932114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10379.158091] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [10382.310066] Lustre: 294896:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.157@tcp [10384.682984] Lustre: server umount MGS complete [10393.723451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10394.278920] LustreError: 295467:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10394.361228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10398.992150] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [10407.717800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10408.215469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10412.601965] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [10424.130591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10424.618640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10431.366087] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [10438.095452] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10440.127883] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [10445.074801] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10446.650815] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10453.485189] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10453.499803] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10453.523832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10453.531649] Lustre: Skipped 4 previous similar messages [10456.775390] Lustre: server umount lustre-OST0000 complete [10464.747071] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10464.756297] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10464.760934] Lustre: Skipped 1 previous similar message [10467.863833] Lustre: server umount lustre-MDT0000 complete [10469.857261] LustreError: 295467:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10469.880278] LustreError: 295467:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10471.980538] LustreError: 295448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775857034 with bad export cookie 7096697787072416238 [10471.983671] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10471.988526] LustreError: 295448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10472.428651] Lustre: server umount lustre-MDT0001 complete [10485.614523] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [10489.025733] Key type lgssc unregistered [10489.372586] LNet: 298649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10489.386596] LNetError: 298649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10490.409822] LNet: Removed LNI 192.168.202.157@tcp [10491.543181] Key type .llcrypt unregistered [10491.544440] Key type ._llcrypt unregistered [10511.652830] Key type ._llcrypt registered [10511.654905] Key type .llcrypt registered [10511.864150] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [10531.599786] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [10532.722785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10532.839575] alg: No test for adler32 (adler32-zlib) [10533.888664] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [10534.122269] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [10535.775203] Key type lgssc registered [10537.042859] Lustre: Echo OBD driver; http://www.lustre.org/ [10578.544938] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10578.586784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10579.937982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10579.973843] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10580.089599] Lustre: lustre-MDT0000: new disk, initializing [10580.260456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10580.284594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10585.034078] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [10596.518382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10596.713906] Lustre: 301774:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10596.794949] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10596.801293] Lustre: Skipped 1 previous similar message [10596.922426] Lustre: lustre-MDT0001: new disk, initializing [10597.079485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10597.142071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10597.159939] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10601.661267] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [10609.688126] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10613.929420] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10623.289719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10623.610823] Lustre: lustre-OST0000: new disk, initializing [10623.615174] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10623.728001] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10630.414556] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [10632.304985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10632.314499] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10632.404860] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10639.612471] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10644.713582] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10644.954178] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10650.873365] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10651.085737] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10658.283255] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10658.283458] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10658.299133] LustreError: Skipped 1 previous similar message [10658.300222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10658.327439] Lustre: Skipped 1 previous similar message [10659.850550] Lustre: server umount lustre-OST0000 complete [10663.904753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10663.916703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10663.930693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10663.933524] Lustre: Skipped 1 previous similar message [10668.512278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10668.514686] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10669.693910] Lustre: server umount lustre-MDT0000 complete [10673.370968] LustreError: 302553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775857235 with bad export cookie 3088164653237670133 [10673.383642] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10673.708324] Lustre: server umount lustre-MDT0001 complete [10683.096815] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 17:40:43 (1775857243) [10699.183055] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [10709.243574] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [10747.381743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10747.664581] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10747.696689] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10747.779910] Lustre: lustre-MDT0000: new disk, initializing [10747.840681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10747.853438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10752.512170] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [10766.229844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10766.423258] Lustre: 308095:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10766.622843] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10766.627883] Lustre: Skipped 1 previous similar message [10766.812064] Lustre: lustre-MDT0001: new disk, initializing [10766.993869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10767.099896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10767.121953] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10772.705939] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [10781.839639] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10786.523750] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10793.983205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10794.423162] Lustre: lustre-OST0000: new disk, initializing [10794.437408] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10794.587102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10801.034945] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [10804.346338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10804.360135] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10804.504436] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10809.630829] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10845.664342] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10845.673324] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10845.682845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10845.683295] Lustre: Skipped 1 previous similar message [10845.696772] Lustre: Skipped 2 previous similar messages [10850.031435] Lustre: server umount lustre-OST0000 complete [10853.789279] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [10853.800168] LustreError: Skipped 1 previous similar message [10853.807449] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10853.822957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10866.144901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10866.150384] Lustre: Skipped 5 previous similar messages [10868.192127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10868.428049] Lustre: server umount lustre-MDT0000 complete [10872.848026] LustreError: 308087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775857435 with bad export cookie 3088164653237671015 [10872.862746] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10872.866692] LustreError: 308087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10873.554786] Lustre: server umount lustre-MDT0001 complete [10884.035971] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [10887.178177] Key type lgssc unregistered [10887.452266] LNet: 311884:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10887.468079] LNetError: 311884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10887.484215] LNet: Removed LNI 192.168.202.157@tcp [10888.280341] Key type .llcrypt unregistered [10888.282792] Key type ._llcrypt unregistered [10976.403311] Key type ._llcrypt registered [10976.404950] Key type .llcrypt registered [10976.514339] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [10976.758306] Key type .llcrypt unregistered [10976.759589] Key type ._llcrypt unregistered [11050.172939] Key type ._llcrypt registered [11050.174872] Key type .llcrypt registered [11050.312029] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [11050.636360] Key type .llcrypt unregistered [11050.637979] Key type ._llcrypt unregistered [11081.892322] Key type ._llcrypt registered [11081.896579] Key type .llcrypt registered [11082.072100] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [11082.466289] Key type .llcrypt unregistered [11082.471255] Key type ._llcrypt unregistered [11112.525387] Key type ._llcrypt registered [11112.526578] Key type .llcrypt registered [11112.614253] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [11112.951343] Key type .llcrypt unregistered [11112.955372] Key type ._llcrypt unregistered [11181.630146] Key type ._llcrypt registered [11181.637355] Key type .llcrypt registered [11181.752258] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [11182.026381] Key type .llcrypt unregistered [11182.029673] Key type ._llcrypt unregistered [11206.409397] Key type ._llcrypt registered [11206.413433] Key type .llcrypt registered [11206.616578] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [11206.913615] Key type .llcrypt unregistered [11206.923445] Key type ._llcrypt unregistered [11230.251787] Key type ._llcrypt registered [11230.253476] Key type .llcrypt registered [11230.334752] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [11230.568384] Key type .llcrypt unregistered [11230.570718] Key type ._llcrypt unregistered [11256.165807] Key type ._llcrypt registered [11256.168462] Key type .llcrypt registered [11256.227617] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [11256.430373] Key type .llcrypt unregistered [11256.432660] Key type ._llcrypt unregistered [11296.472942] Key type ._llcrypt registered [11296.475870] Key type .llcrypt registered [11296.596626] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [11296.888316] Key type .llcrypt unregistered [11296.891100] Key type ._llcrypt unregistered [11329.359561] Key type ._llcrypt registered [11329.361480] Key type .llcrypt registered [11329.445119] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [11329.716654] Key type .llcrypt unregistered [11329.719245] Key type ._llcrypt unregistered [11396.829340] Key type ._llcrypt registered [11396.833674] Key type .llcrypt registered [11396.935036] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [11397.212462] Key type .llcrypt unregistered [11397.214208] Key type ._llcrypt unregistered [11425.695470] Key type ._llcrypt registered [11425.700587] Key type .llcrypt registered [11425.814907] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [11426.101336] Key type .llcrypt unregistered [11426.103976] Key type ._llcrypt unregistered [11458.180421] Key type ._llcrypt registered [11458.182120] Key type .llcrypt registered [11458.312827] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [11458.688199] Key type .llcrypt unregistered [11458.689820] Key type ._llcrypt unregistered [11514.109875] Key type ._llcrypt registered [11514.112173] Key type .llcrypt registered [11514.254512] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [11530.368656] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [11531.956340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11532.147397] alg: No test for adler32 (adler32-zlib) [11533.210856] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [11533.433933] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [11535.114638] Key type lgssc registered [11536.374262] Lustre: Echo OBD driver; http://www.lustre.org/ [11575.781196] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11575.854633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11577.371837] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [11577.449470] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [11577.661441] Lustre: tst250fs-MDT0000: new disk, initializing [11577.878202] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11577.904470] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11583.563885] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [11598.874351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11599.022794] Lustre: 331865:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11599.089559] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [11599.092911] Lustre: Skipped 1 previous similar message [11599.179688] Lustre: tst250fs-MDT0001: new disk, initializing [11599.271063] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11599.310192] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11599.315584] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11603.633258] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [11609.976107] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11618.806552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11618.970314] Lustre: tst250fs-OST0000: new disk, initializing [11618.973175] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [11619.078408] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11623.564203] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11623.577039] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11623.758731] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11625.974450] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [11637.234563] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11637.448355] Lustre: tst250fs-OST0001: new disk, initializing [11637.454427] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [11637.625157] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11643.572063] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11643.587935] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11643.723389] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11645.020303] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [11651.152896] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [11653.269331] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [11653.273923] Lustre: Skipped 1 previous similar message [11674.604766] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11674.607452] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [11674.622467] Lustre: Skipped 3 previous similar messages [11674.648573] Lustre: Skipped 3 previous similar messages [11679.730622] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [11679.747463] Lustre: Skipped 3 previous similar messages [11680.824309] Lustre: server umount tst250fs-MDT0000 complete [11684.019973] LustreError: 331857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775858246 with bad export cookie 14512610248100838365 [11684.030398] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11684.034411] LustreError: 331857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11684.469288] Lustre: server umount tst250fs-MDT0001 complete [11698.423228] Lustre: server umount tst250fs-OST0000 complete [11701.226824] Lustre: 330090:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775858247/real 1775858247] req@ffff8faeb6796d80 x1862122177895680/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775858263 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11701.251264] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11702.292942] Lustre: server umount tst250fs-OST0001 complete [11724.015264] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [11736.347940] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [11772.949648] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 17:58:51 (1775858331) [11775.862827] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [11795.792584] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [11807.075364] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [11841.175613] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11600 sec ========= 18:00:01 (1775858401) [11842.887868] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:00:03 (1775858403) === [11846.991777] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:00:06 (1775858406) === [11895.622314] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [11899.328904] Key type lgssc unregistered [11899.984478] LNet: 343377:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11899.998049] LNetError: 343377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11900.024215] LNet: Removed LNI 192.168.202.157@tcp [11901.180388] Key type .llcrypt unregistered [11901.184305] Key type ._llcrypt unregistered