[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 406509265 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002330] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.007268] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008016] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009006] pid_max: default: 32768 minimum: 301 [ 0.010115] LSM: Security Framework initializing [ 0.011052] Yama: becoming mindful. [ 0.012028] SELinux: Initializing. [ 0.013051] *** VALIDATE selinux *** [ 0.021541] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025148] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026089] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027077] *** VALIDATE tmpfs *** [ 0.028333] *** VALIDATE proc *** [ 0.029157] *** VALIDATE cgroup *** [ 0.030006] *** VALIDATE cgroup2 *** [ 0.031210] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032113] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034025] Spectre V2 : User space: Vulnerable [ 0.035007] Speculative Store Bypass: Vulnerable [ 0.037979] debug: unmapping init [mem 0xffffffffb7e59000-0xffffffffb7e60fff] [ 0.040194] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041521] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042015] ... version: 2 [ 0.042813] ... bit width: 48 [ 0.043010] ... generic registers: 4 [ 0.044007] ... value mask: 0000ffffffffffff [ 0.045007] ... max period: 00007fffffffffff [ 0.046009] ... fixed-purpose events: 3 [ 0.047008] ... event mask: 000000070000000f [ 0.048287] rcu: Hierarchical SRCU implementation. [ 0.050364] smp: Bringing up secondary CPUs ... [ 0.051515] x86: Booting SMP configuration: [ 0.052015] .... node #0, CPUs: #1 #2 #3 [ 0.054705] smp: Brought up 1 node, 4 CPUs [ 0.056008] smpboot: Max logical packages: 1 [ 0.057008] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.135018] node 0 deferred pages initialised in 77ms [ 0.137131] devtmpfs: initialized [ 0.138170] x86/mm: Memory block size: 128MB [ 0.140678] gcov: version magic: 0x41383552 [ 0.142137] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.145088] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.147267] pinctrl core: initialized pinctrl subsystem [ 0.148081] [ 0.148464] ************************************************************* [ 0.151010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.154009] ** ** [ 0.156008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.158009] ** ** [ 0.161009] ** This means that this kernel is built to expose internal ** [ 0.162005] ** IOMMU data structures, which may compromise security on ** [ 0.164007] ** your system. ** [ 0.166008] ** ** [ 0.168006] ** If you see this message and you are not debugging the ** [ 0.169006] ** kernel, report this immediately to your vendor! ** [ 0.171008] ** ** [ 0.173007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.175010] ************************************************************* [ 0.177537] NET: Registered protocol family 16 [ 0.178328] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.180035] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.183034] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.186103] cpuidle: using governor menu [ 0.187459] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.189431] PCI: Using configuration type 1 for base access [ 0.192122] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.197097] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.199023] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.201083] cryptd: max_cpu_qlen set to 1000 [ 0.203155] ACPI: Added _OSI(Module Device) [ 0.204070] ACPI: Added _OSI(Processor Device) [ 0.205008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.207009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.211859] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.216366] ACPI: Interpreter enabled [ 0.218046] ACPI: PM: (supports S0 S3 S4 S5) [ 0.219006] ACPI: Using IOAPIC for interrupt routing [ 0.220050] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.222218] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.229759] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.231015] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.232007] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.234032] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.236861] acpiphp: Slot [2] registered [ 0.238076] acpiphp: Slot [5] registered [ 0.238861] acpiphp: Slot [6] registered [ 0.240047] acpiphp: Slot [7] registered [ 0.240751] acpiphp: Slot [8] registered [ 0.241042] acpiphp: Slot [9] registered [ 0.241874] acpiphp: Slot [10] registered [ 0.243045] acpiphp: Slot [3] registered [ 0.243871] acpiphp: Slot [4] registered [ 0.245052] acpiphp: Slot [11] registered [ 0.245890] acpiphp: Slot [12] registered [ 0.246037] acpiphp: Slot [13] registered [ 0.246848] acpiphp: Slot [14] registered [ 0.248037] acpiphp: Slot [15] registered [ 0.248799] acpiphp: Slot [16] registered [ 0.249036] acpiphp: Slot [17] registered [ 0.249832] acpiphp: Slot [18] registered [ 0.251037] acpiphp: Slot [19] registered [ 0.251794] acpiphp: Slot [20] registered [ 0.252044] acpiphp: Slot [21] registered [ 0.252824] acpiphp: Slot [22] registered [ 0.254038] acpiphp: Slot [23] registered [ 0.254852] acpiphp: Slot [24] registered [ 0.256036] acpiphp: Slot [25] registered [ 0.256934] acpiphp: Slot [26] registered [ 0.257049] acpiphp: Slot [27] registered [ 0.257826] acpiphp: Slot [28] registered [ 0.259037] acpiphp: Slot [29] registered [ 0.259847] acpiphp: Slot [30] registered [ 0.261038] acpiphp: Slot [31] registered [ 0.261753] PCI host bridge to bus 0000:00 [ 0.262020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.263007] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.265008] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.266007] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.268007] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.270013] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.271138] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.272632] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.275123] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.285013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.288795] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.291010] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.293011] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.295009] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.297378] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.299530] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.302026] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.303495] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.306685] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.317011] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.320006] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.324017] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.329010] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.334009] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.346013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.351877] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.357011] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.361010] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.374015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.382047] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.386013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.390012] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.401011] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.409485] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.415010] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.420010] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.429010] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.435590] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.439010] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.443009] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.454014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.461783] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.467013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.472014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.488023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.497176] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.499313] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.501194] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.502274] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.504147] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.509031] iommu: Default domain type: Passthrough [ 0.510295] SCSI subsystem initialized [ 0.511070] ACPI: bus type USB registered [ 0.511976] usbcore: registered new interface driver usbfs [ 0.513028] usbcore: registered new interface driver hub [ 0.514031] usbcore: registered new device driver usb [ 0.515095] pps_core: LinuxPPS API ver. 1 registered [ 0.517008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.520036] PTP clock support registered [ 0.522066] EDAC MC: Ver: 3.0.0 [ 0.524136] PCI: Using ACPI for IRQ routing [ 0.525812] NetLabel: Initializing [ 0.527008] NetLabel: domain hash size = 128 [ 0.529020] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.531076] NetLabel: unlabeled traffic allowed by default [ 0.533117] vgaarb: loaded [ 0.534179] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.535010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.542247] clocksource: Switched to clocksource kvm-clock [ 0.636782] VFS: Disk quotas dquot_6.6.0 [ 0.638238] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.640291] *** VALIDATE ramfs *** [ 0.640995] *** VALIDATE hugetlbfs *** [ 0.642092] pnp: PnP ACPI init [ 0.644501] pnp: PnP ACPI: found 6 devices [ 0.658800] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.662315] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.664496] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.666917] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.669519] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.671108] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.673394] NET: Registered protocol family 2 [ 0.675929] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.680554] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.683155] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.688144] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.691134] TCP: Hash tables configured (established 65536 bind 65536) [ 0.693413] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.695362] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.697739] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.700700] NET: Registered protocol family 1 [ 0.703699] RPC: Registered named UNIX socket transport module. [ 0.705847] RPC: Registered udp transport module. [ 0.707498] RPC: Registered tcp transport module. [ 0.708625] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.710099] NET: Registered protocol family 44 [ 0.711141] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.712558] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.713864] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.715449] PCI: CLS 0 bytes, default 64 [ 0.716941] Unpacking initramfs... [ 1.955727] debug: unmapping init [mem 0xffff9971fcc54000-0xffff9971fffbffff] [ 1.959139] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.961130] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.963743] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.553309] Initialise system trusted keyrings [ 2.554899] Key type blacklist registered [ 2.556693] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.563203] zbud: loaded [ 2.565416] *** VALIDATE nfs *** [ 2.566052] *** VALIDATE nfs4 *** [ 2.566944] pstore: using deflate compression [ 2.568752] Platform Keyring initialized [ 2.637720] NET: Registered protocol family 38 [ 2.639271] Key type asymmetric registered [ 2.640892] Asymmetric key parser 'x509' registered [ 2.643050] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.645732] io scheduler mq-deadline registered [ 2.647267] io scheduler kyber registered [ 2.648756] io scheduler bfq registered [ 2.650280] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.652095] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.653837] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.656229] ACPI: Power Button [PWRF] [ 2.661551] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.667498] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.674064] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.677810] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.685541] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.711809] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.738256] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.741612] Non-volatile memory driver v1.3 [ 2.742527] Linux agpgart interface v0.103 [ 2.762903] virtio_blk virtio1: [vda] 134144 512-byte logical blocks (68.7 MB/65.5 MiB) [ 2.764648] vda: detected capacity change from 0 to 68681728 [ 2.774788] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.777434] vdb: detected capacity change from 0 to 1073741824 [ 2.787849] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.789764] vdc: detected capacity change from 0 to 2621440000 [ 2.799723] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.801693] vdd: detected capacity change from 0 to 2621440000 [ 2.811802] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.813619] vde: detected capacity change from 0 to 4294967296 [ 2.823048] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.824785] vdf: detected capacity change from 0 to 4294967296 [ 2.828386] libphy: Fixed MDIO Bus: probed [ 2.831468] usbcore: registered new interface driver usbserial_generic [ 2.833357] usbserial: USB Serial support registered for generic [ 2.834891] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.837443] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.839370] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.841592] mousedev: PS/2 mouse device common for all mice [ 2.844075] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.845322] rtc_cmos 00:05: RTC can wake from S4 [ 2.848101] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.849062] rtc_cmos 00:05: registered as rtc0 [ 2.851090] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.851314] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.855278] intel_pstate: CPU model not supported [ 2.857540] hid: raw HID events driver (C) Jiri Kosina [ 2.858861] usbcore: registered new interface driver usbhid [ 2.860394] usbhid: USB HID core driver [ 2.861503] drop_monitor: Initializing network drop monitor service [ 2.863131] Initializing XFRM netlink socket [ 2.864602] NET: Registered protocol family 10 [ 2.866604] Segment Routing with IPv6 [ 2.867525] NET: Registered protocol family 17 [ 2.868798] mpls_gso: MPLS GSO support [ 2.873727] RAS: Correctable Errors collector initialized. [ 2.875035] AVX version of gcm_enc/dec engaged. [ 2.875910] AES CTR mode by8 optimization enabled [ 2.914994] sched_clock: Marking stable (2914975327, 0)->(3579053238, -664077911) [ 2.918341] registered taskstats version 1 [ 2.919519] Loading compiled-in X.509 certificates [ 2.920964] zswap: loaded using pool lzo/zbud [ 2.935711] Key type big_key registered [ 2.943179] Key type encrypted registered [ 2.944147] ima: No TPM chip found, activating TPM-bypass! [ 2.945270] ima: Allocated hash algorithm: sha1 [ 2.946141] ima: No architecture policies found [ 2.947398] evm: Initialising EVM extended attributes: [ 2.949375] evm: security.selinux [ 2.950567] evm: security.ima [ 2.951645] evm: security.capability [ 2.952469] evm: HMAC attrs: 0x1 [ 2.953751] rtc_cmos 00:05: setting system clock to 2026-05-09 17:21:36 UTC (1778347296) [ 2.957337] debug: unmapping init [mem 0xffffffffb8e03000-0xffffffffb8ffffff] [ 2.958813] debug: unmapping init [mem 0xffffffffb7b82000-0xffffffffb7e58fff] [ 2.966047] Write protecting the kernel read-only data: 28672k [ 2.968103] debug: unmapping init [mem 0xffffffffb6203000-0xffffffffb63fffff] [ 2.969724] debug: unmapping init [mem 0xffffffffb6b14000-0xffffffffb6bfffff] [ 2.992821] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.998936] systemd[1]: Detected virtualization kvm. [ 3.000023] systemd[1]: Detected architecture x86-64. [ 3.001106] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.024020] systemd[1]: No hostname configured. [ 3.025325] systemd[1]: Set hostname to . [ 3.026647] random: systemd: uninitialized urandom read (16 bytes read) [ 3.028116] systemd[1]: Initializing machine ID from random generator. [ 3.056981] random: ln: uninitialized urandom read (6 bytes read) [ 3.121549] random: systemd: uninitialized urandom read (16 bytes read) [ 3.123127] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.125896] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.128103] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.522836] device-mapper: uevent: version 1.0.3 [ 3.524200] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] [ 4.112540] virtio_net virtio0 ens2: renamed from eth0 Started Hardware RNG Entropy Gatherer Daemon. [ 4.132386] random: fast init done [ 4.188419] scsi host0: ata_piix [ 4.207185] scsi host1: ata_piix [ 4.217742] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.220225] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.414082] dracut-initqueue[598]: RTNETLINK answers: File exists [ 8.456708] random: crng init done [ 8.457791] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 8.858439] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 9.679233] printk: systemd: 26 output lines suppressed due to ratelimiting [ 9.848216] SELinux: Disabled at runtime. [ 9.895707] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 9.899972] systemd[1]: Detected virtualization kvm. [ 9.901830] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 10.246962] systemd[1]: initrd-switch-root.service: Succeeded. [ 10.249516] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 10.253919] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 10.257896] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 10.260552] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 10.269033] systemd[1]: Starting Journal Service... Starting Journal Service... [ 10.272160] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [ 10.371351] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Mounted /mnt. [ 10.538848] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 10.805384] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 10.823123] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 10.913346] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 10.919697] EDAC sbridge: Ver: 1.1.2 [ 11.944249] Key type dns_resolver registered [ 12.184018] NFS: Registering the id_resolver key type [ 12.185069] Key type id_resolver registered [ 12.185826] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg213-server login: [ 33.591106] libcfs: loading out-of-tree module taints kernel. [ 33.603840] Key type ._llcrypt registered [ 33.605249] Key type .llcrypt registered [ 33.639893] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 40.954658] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 41.679218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 41.687821] alg: No test for adler32 (adler32-zlib) [ 42.722338] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 43.040430] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 44.656715] Key type lgssc registered [ 45.342096] Lustre: Echo OBD driver; http://www.lustre.org/ [ 52.170234] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 66.325078] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 70.828274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 70.836693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 71.925597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 71.940662] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 71.980474] Lustre: lustre-MDT0000: new disk, initializing [ 72.010467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 72.018731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 73.420544] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 78.543741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 78.579617] Lustre: 6486:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 78.592886] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 78.595063] Lustre: Skipped 1 previous similar message [ 78.632192] Lustre: lustre-MDT0001: new disk, initializing [ 78.652623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 78.660875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 78.663992] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 79.857817] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 82.019808] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 85.025226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 85.107935] Lustre: lustre-OST0000: new disk, initializing [ 85.109460] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 85.126663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 86.858358] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 88.235113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 88.238938] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 88.259341] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 91.966870] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 92.004847] Lustre: lustre-OST0001: new disk, initializing [ 92.007534] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 92.024458] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 93.949753] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 96.745539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 96.748068] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 96.761546] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 99.575869] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 104.095036] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 111.080249] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing check_logdir /tmp/testlogs/ [ 112.427307] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing yml_node [ 113.756364] Lustre: DEBUG MARKER: Client: 2.17.52.4 [ 114.474562] Lustre: DEBUG MARKER: MDS: 2.17.52.4 [ 115.198849] Lustre: DEBUG MARKER: OSS: 2.17.52.4 [ 115.690328] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat May 9 13:23:27 EDT 2026 [ 121.582248] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 122.131914] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 135.135631] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 135.138330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 135.142074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 137.696192] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 137.696311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 137.700785] Lustre: Skipped 1 previous similar message [ 137.703578] Lustre: Skipped 1 previous similar message [ 140.060727] Lustre: server umount lustre-MDT0000 complete [ 141.317962] LustreError: 6478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778347434 with bad export cookie 7504302425488318590 [ 141.319865] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 141.321400] LustreError: 6478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 142.815823] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 142.816091] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 142.820855] Lustre: Skipped 2 previous similar messages [ 146.847742] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 146.849513] Lustre: Skipped 2 previous similar messages [ 147.498263] Lustre: server umount lustre-MDT0001 complete [ 154.743977] Lustre: server umount lustre-OST0000 complete [ 161.998693] Lustre: server umount lustre-OST0001 complete [ 164.028118] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 167.348973] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 179.747158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 179.835317] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 179.848063] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 179.885853] Lustre: lustre-MDT0000: new disk, initializing [ 179.916543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 179.926597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 181.246654] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 185.460450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 185.505774] Lustre: 14951:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 185.614969] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 186.883919] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 189.935174] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 191.133273] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 193.437449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 193.550296] Lustre: lustre-OST0000: new disk, initializing [ 193.552647] Lustre: Skipped 1 previous similar message [ 193.555390] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 193.558652] Lustre: Skipped 2 previous similar messages [ 193.589321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 193.591695] Lustre: Skipped 1 previous similar message [ 195.257517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 195.259854] Lustre: Skipped 1 previous similar message [ 195.261865] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 195.298255] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 195.403861] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 198.430174] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 199.853188] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 199.942071] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 201.397273] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 201.475317] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 205.794213] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 205.794903] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 205.800487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 205.801713] LustreError: Skipped 1 previous similar message [ 205.806704] Lustre: Skipped 1 previous similar message [ 208.274978] Lustre: server umount lustre-OST0000 complete [ 210.912624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 210.916025] Lustre: Skipped 2 previous similar messages [ 215.643642] Lustre: server umount lustre-MDT0000 complete [ 216.032347] LustreError: 14961:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 216.037501] LustreError: 14961:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 216.926304] LustreError: 14943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778347510 with bad export cookie 7504302425488320109 [ 216.929169] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 216.929890] LustreError: 14943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 217.048137] Lustre: server umount lustre-MDT0001 complete [ 218.849490] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 13:25:10 (1778347510) [ 225.033551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 225.178307] LustreError: 19418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 225.200459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 226.569249] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 229.507956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 230.995613] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 232.817647] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 233.965173] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 234.697720] Lustre: Failing over lustre-MDT0000 [ 234.701399] LustreError: 19448:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 234.704234] LustreError: 19448:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 234.706567] LustreError: 19448:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -5 [ 234.805432] Lustre: server umount lustre-MDT0000 complete [ 234.975720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 234.976598] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 234.978308] LustreError: 19419:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 234.981216] Lustre: Skipped 1 previous similar message [ 240.096220] LustreError: 20127:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 240.101393] LustreError: 20127:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 245.216119] LustreError: 19418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 245.221561] LustreError: 19418:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 247.811040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 247.851309] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 249.226122] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 252.329740] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 253.408271] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 253.410651] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 253.416806] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 255.392514] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 255.393535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 255.396299] Lustre: Skipped 1 previous similar message [ 255.399970] Lustre: Skipped 3 previous similar messages [ 260.498988] Lustre: server umount lustre-MDT0000 complete [ 260.576112] LustreError: 19418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 260.581940] LustreError: 19418:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 261.720087] LustreError: 19398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778347555 with bad export cookie 7504302425488321516 [ 261.723345] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 261.724117] LustreError: 19398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 261.833436] Lustre: server umount lustre-MDT0001 complete [ 264.619990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 264.794969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 264.797485] Lustre: Skipped 2 previous similar messages [ 266.100760] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 268.971541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 270.435341] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 272.295228] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 273.571954] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 274.301577] Lustre: Failing over lustre-MDT0000 [ 274.304880] LustreError: 22440:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 274.307644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 274.308422] LustreError: 22440:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 274.309804] Lustre: Skipped 1 previous similar message [ 274.314180] LustreError: 22440:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 274.404950] Lustre: server umount lustre-MDT0000 complete [ 279.456273] LustreError: 22411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 279.460918] LustreError: 22411:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 287.349450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 287.388465] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 288.773227] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 291.923524] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 292.835797] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 292.839321] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 292.840977] Lustre: Skipped 1 previous similar message [ 292.846070] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 294.944814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 294.948717] Lustre: Skipped 2 previous similar messages [ 308.191164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 308.258421] Lustre: server umount lustre-MDT0000 complete [ 309.556691] LustreError: 24728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778347603 with bad export cookie 7504302425488322447 [ 309.559932] LustreError: 24728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 309.560121] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 309.670821] Lustre: server umount lustre-MDT0001 complete [ 312.562861] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 13:26:44 (1778347604) [ 318.031589] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 320.959436] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 333.787435] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 337.183136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 337.260888] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 337.271601] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 337.305953] Lustre: lustre-MDT0000: new disk, initializing [ 337.328077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 337.329864] Lustre: Skipped 2 previous similar messages [ 337.334078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 338.486887] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 342.376262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 342.409488] Lustre: 28383:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 342.505568] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 343.717338] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 346.628489] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 347.807871] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 349.956777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 351.424741] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 351.448300] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 351.454541] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 352.014604] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 355.117016] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 357.064864] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 361.954994] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 361.957053] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 361.957576] LustreError: Skipped 1 previous similar message [ 361.961628] Lustre: Skipped 1 previous similar message [ 361.964218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 361.967072] Lustre: Skipped 7 previous similar messages [ 366.900735] Lustre: server umount lustre-OST0000 complete [ 374.369960] Lustre: server umount lustre-MDT0000 complete [ 375.671395] LustreError: 28373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778347669 with bad export cookie 7504302425488322853 [ 375.674080] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 375.675345] LustreError: 28373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 375.844190] Lustre: server umount lustre-MDT0001 complete [ 379.513559] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 380.588889] Key type lgssc unregistered [ 380.723434] LNet: 31945:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 380.726337] LNetError: 31945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 380.736362] LNet: Removed LNI 192.168.202.113@tcp [ 381.061121] Key type .llcrypt unregistered [ 381.062211] Key type ._llcrypt unregistered [ 390.765764] Key type ._llcrypt registered [ 390.766719] Key type .llcrypt registered [ 390.802911] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 391.179950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 391.215829] alg: No test for adler32 (adler32-zlib) [ 392.084399] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 392.179571] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 393.767127] Key type lgssc registered [ 394.172820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 396.832967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 396.841405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 398.022437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 399.361238] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 402.240573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 402.364197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 403.690120] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 405.548073] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 406.749356] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 408.973918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 409.108173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 411.057936] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 412.205417] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 412.860512] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 433.233945] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 13:28:45 (1778347725) [ 433.843459] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 434.444143] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 436.579215] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 13:28:48 (1778347728) [ 439.504975] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 13:28:51 (1778347731) [ 439.867189] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 442.606097] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 442.609273] Lustre: Skipped 2 previous similar messages [ 443.165789] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 13:28:55 (1778347735) [ 450.163621] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 450.166606] Lustre: Skipped 10 previous similar messages [ 451.029455] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 13:29:03 (1778347743) [ 451.952856] Lustre: *** cfs_fail_loc=131b, val=2*** [ 453.160499] Lustre: *** cfs_fail_loc=131b, val=2*** [ 455.878828] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 13:29:07 (1778347747) [ 460.177840] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 460.181633] Lustre: Skipped 7 previous similar messages [ 460.784080] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 13:29:12 (1778347752) [ 468.447891] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 468.450575] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 468.454495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 469.474902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 469.475366] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 469.475527] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 470.691372] Lustre: server umount lustre-OST0000 complete [ 474.079859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 474.081992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 474.085455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 474.087010] Lustre: Skipped 1 previous similar message [ 479.711956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 479.714703] Lustre: Skipped 2 previous similar messages [ 484.832893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 484.835478] Lustre: Skipped 1 previous similar message [ 486.367140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 486.428172] Lustre: server umount lustre-MDT0000 complete [ 487.674364] LustreError: 33297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778347781 with bad export cookie 17394932818314077116 [ 487.676649] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 487.677426] LustreError: 33297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 487.790534] Lustre: server umount lustre-MDT0001 complete [ 491.438290] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 492.357929] Key type lgssc unregistered [ 492.475402] LNet: 40818:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 492.477641] LNetError: 40818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 492.485292] LNet: Removed LNI 192.168.202.113@tcp [ 492.777919] Key type .llcrypt unregistered [ 492.778937] Key type ._llcrypt unregistered [ 498.502780] Key type ._llcrypt registered [ 498.503871] Key type .llcrypt registered [ 498.540352] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 504.816494] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 505.115761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 505.163069] alg: No test for adler32 (adler32-zlib) [ 506.011728] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 506.087955] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 507.663228] Key type lgssc registered [ 508.010305] Lustre: Echo OBD driver; http://www.lustre.org/ [ 519.091093] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 519.096555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 520.192460] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 520.203248] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 520.241775] Lustre: lustre-MDT0000: new disk, initializing [ 520.267586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 520.273782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 521.586303] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 525.812760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 525.856264] Lustre: 43909:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 525.881639] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 525.883622] Lustre: Skipped 1 previous similar message [ 525.923269] Lustre: lustre-MDT0001: new disk, initializing [ 525.956169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 525.969062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 525.973336] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 527.298082] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 530.369570] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 531.520734] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 533.784820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 533.900157] Lustre: lustre-OST0000: new disk, initializing [ 533.902468] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 533.930170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 535.877527] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 538.983300] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 539.127852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 539.131822] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 539.164036] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 540.402213] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 540.489123] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 541.827159] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 541.904366] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 544.226141] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 544.226215] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 544.227368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 544.228862] LustreError: Skipped 1 previous similar message [ 544.234440] Lustre: Skipped 1 previous similar message [ 548.693836] Lustre: server umount lustre-OST0000 complete [ 551.903753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 551.906045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 551.909777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 551.911527] Lustre: Skipped 1 previous similar message [ 554.464136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 554.464542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 556.117795] Lustre: server umount lustre-MDT0000 complete [ 557.233192] LustreError: 43900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778347850 with bad export cookie 2249346942270092880 [ 557.235374] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 557.237762] LustreError: 43900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 557.339771] Lustre: server umount lustre-MDT0001 complete [ 560.335756] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 13:30:52 (1778347852) [ 562.421246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 562.574625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 563.727884] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 566.310048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 567.696167] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 569.489105] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 570.718908] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 572.992439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 573.102219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 573.104590] Lustre: Skipped 1 previous similar message [ 575.051042] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 576.870326] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 594.199748] LustreError: 50173:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 594.510546] Lustre: Modifying parameter general.timeout=1 in log params [ 595.708787] Lustre: Modifying parameter general.timeout=5 in log params [ 595.711371] Lustre: Skipped 3 previous similar messages [ 597.818897] Lustre: Modifying parameter general.timeout=12 in log params [ 597.820275] Lustre: Skipped 6 previous similar messages [ 601.972574] Lustre: Modifying parameter general.timeout=26 in log params [ 601.974783] Lustre: Skipped 13 previous similar messages [ 610.268593] Lustre: Modifying parameter general.timeout=54 in log params [ 610.270036] Lustre: Skipped 27 previous similar messages [ 626.548961] Lustre: Modifying parameter general.timeout=109 in log params [ 626.550566] Lustre: Skipped 54 previous similar messages [ 636.100439] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 13:32:08 (1778347928) [ 637.085911] Lustre: 57075:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} minrate=50 rate=500 (mode = 2) failed: rc = -17 [ 638.274547] Lustre: 57273:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 638.278404] Lustre: 57273:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 640.639262] Lustre: 57666:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 minrate=50 rate=5000 (mode = 2) failed: rc = -17 [ 640.645495] Lustre: 57666:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 3 previous similar messages [ 650.341270] Lustre: 59166:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 650.346830] LustreError: 59166:0:(mgs_handler.c:1164:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 651.746520] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 13:32:23 (1778347943) [ 657.619470] Lustre: server umount lustre-MDT0000 complete [ 658.402315] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 658.404389] LustreError: 50046:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 658.405806] Lustre: Skipped 2 previous similar messages [ 658.411697] LustreError: 50046:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 658.839224] LustreError: 47939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778347952 with bad export cookie 2249346942270093748 [ 658.841449] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 658.843029] LustreError: 47939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 663.520038] LustreError: 47954:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 663.520273] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 663.533615] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 663.535465] Lustre: Skipped 1 previous similar message [ 664.999339] Lustre: server umount lustre-MDT0001 complete [ 681.473450] Lustre: server umount lustre-OST0000 complete [ 690.021394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 690.108521] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 690.117764] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 690.122133] Lustre: Skipped 46 previous similar messages [ 690.204459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 691.474191] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 694.217899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 694.249609] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 694.258869] Lustre: 61695:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 695.621207] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 697.426809] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 698.638929] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 700.884123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 700.948042] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 702.881714] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 704.611333] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 709.090305] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 709.090828] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 709.091841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 709.092783] LustreError: Skipped 1 previous similar message [ 709.095827] Lustre: Skipped 1 previous similar message [ 713.103462] Lustre: server umount lustre-OST0000 complete [ 714.719741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 719.328772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 719.331513] Lustre: Skipped 3 previous similar messages [ 720.476233] Lustre: server umount lustre-MDT0000 complete [ 721.699021] LustreError: 61687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778348015 with bad export cookie 2249346942270096044 [ 721.702427] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 721.702612] LustreError: 61687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 721.816048] Lustre: server umount lustre-MDT0001 complete [ 725.859317] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 726.850879] Key type lgssc unregistered [ 726.985420] LNet: 65106:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 726.987868] LNetError: 65106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 726.999407] LNet: Removed LNI 192.168.202.113@tcp [ 727.306054] Key type .llcrypt unregistered [ 727.306992] Key type ._llcrypt unregistered [ 737.042687] Key type ._llcrypt registered [ 737.043763] Key type .llcrypt registered [ 737.080783] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 737.416017] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 737.437053] alg: No test for adler32 (adler32-zlib) [ 738.291062] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 738.368867] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 739.951226] Key type lgssc registered [ 740.333526] Lustre: Echo OBD driver; http://www.lustre.org/ [ 742.955634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 742.961503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 744.128125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 745.382655] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 748.459158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 748.653106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 750.202110] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 752.438788] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 754.124371] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 756.629511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 756.804843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 758.989783] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 761.050597] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 774.249642] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 779.231789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 779.233797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 779.237079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 779.745352] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 779.746600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 779.748481] Lustre: Skipped 1 previous similar message [ 779.751417] Lustre: Skipped 1 previous similar message [ 782.809591] Lustre: server umount lustre-MDT0000 complete [ 783.991057] LustreError: 68845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778348077 with bad export cookie 17680513151948968697 [ 783.994541] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 783.997129] LustreError: 68845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 784.104455] Lustre: server umount lustre-MDT0001 complete [ 794.585104] Lustre: server umount lustre-OST0000 complete [ 803.104550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 803.196116] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 803.205092] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 803.297641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 804.539429] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 807.286329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 807.319037] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 807.328266] Lustre: 71034:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 807.479822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 808.730199] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 810.509465] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 811.660466] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 813.940090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 814.013160] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 816.181853] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 818.085359] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 823.701424] Lustre: 73266:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 824.370760] Lustre: Modifying parameter general.jobid_var=disable in log params [ 829.410954] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 829.411386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 829.415108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 829.420223] Lustre: Skipped 1 previous similar message [ 829.426815] Lustre: Skipped 1 previous similar message [ 831.689402] Lustre: server umount lustre-OST0000 complete [ 832.991990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 832.994166] LustreError: Skipped 1 previous similar message [ 832.995490] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 832.999190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 839.648604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 839.650393] Lustre: Skipped 3 previous similar messages [ 847.327153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 847.404763] Lustre: server umount lustre-MDT0000 complete [ 848.585146] LustreError: 73269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778348142 with bad export cookie 17680513151948969516 [ 848.587765] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 848.589819] LustreError: 73269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 848.700417] Lustre: server umount lustre-MDT0001 complete [ 852.152442] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 853.135433] Key type lgssc unregistered [ 853.262323] LNet: 74461:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 853.264420] LNetError: 74461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 853.275269] LNet: Removed LNI 192.168.202.113@tcp [ 853.556435] Key type .llcrypt unregistered [ 853.557437] Key type ._llcrypt unregistered [ 862.652578] Key type ._llcrypt registered [ 862.653565] Key type .llcrypt registered [ 862.687504] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 862.997941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 863.016055] alg: No test for adler32 (adler32-zlib) [ 863.864921] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 863.941148] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 865.519090] Key type lgssc registered [ 865.859309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 868.640359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 868.646336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 869.800487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 871.034550] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 873.604832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 873.781542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 875.396600] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 877.257228] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 878.347332] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 880.583603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 880.775627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 882.606518] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 884.373114] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 902.310849] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 903.007512] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 903.009325] Lustre: Skipped 1 previous similar message [ 904.267350] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 904.269133] Lustre: Skipped 3 previous similar messages [ 906.472674] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 906.474478] Lustre: Skipped 5 previous similar messages [ 910.567924] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 910.571537] Lustre: Skipped 13 previous similar messages [ 918.676996] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 918.678782] Lustre: Skipped 27 previous similar messages [ 935.097107] Lustre: 83584:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 935.396427] Lustre: Disabling parameter general.jobid_var= in log params [ 935.398134] Lustre: Skipped 45 previous similar messages [ 935.933597] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 13:37:08 (1778348228) [ 936.424294] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 936.998827] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 13:37:09 (1778348229) [ 940.513431] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 940.517277] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 940.523395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 941.536856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 941.537425] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 941.538290] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 946.656964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 946.658762] Lustre: Skipped 2 previous similar messages [ 951.778855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 951.782019] Lustre: Skipped 1 previous similar message [ 952.287099] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 952.335519] Lustre: server umount lustre-OST0000 complete [ 955.872179] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 955.875870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 955.882268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 959.697811] Lustre: server umount lustre-MDT0000 complete [ 960.893913] LustreError: 79191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778348254 with bad export cookie 14140214800812930282 [ 960.896808] LustreError: 79191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 960.897183] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 961.008454] Lustre: server umount lustre-MDT0001 complete [ 964.424794] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 965.420462] Key type lgssc unregistered [ 965.542384] LNet: 84814:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.545335] LNetError: 84814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 965.552316] LNet: Removed LNI 192.168.202.113@tcp [ 965.862062] Key type .llcrypt unregistered [ 965.863289] Key type ._llcrypt unregistered [ 967.380472] Key type ._llcrypt registered [ 967.381809] Key type .llcrypt registered [ 967.425783] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 968.853650] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules [ 969.256898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 969.269481] alg: No test for adler32 (adler32-zlib) [ 970.119358] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 970.193076] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 971.767161] Key type lgssc registered [ 972.140554] Lustre: Echo OBD driver; http://www.lustre.org/ [ 972.509114] LustreError: 85935:0:(lu_object.c:1477:lu_context_key_register()) cfs_race id 60d sleeping [ 980.214638] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 983.215343] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 993.793641] LustreError: 85935:0:(lu_object.c:1477:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 993.802421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 994.995292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 996.284260] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 999.064706] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 13:38:11 (1778348291) [ 1000.603723] LustreError: 86971:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1000.608096] LustreError: 86971:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1000.611907] LustreError: 86971:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 1000.734416] Lustre: server umount lustre-MDT0000 complete [ 1005.046974] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 1005.991757] Key type lgssc unregistered [ 1006.108393] LNet: 88428:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1006.111460] LNetError: 88428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1006.121327] LNet: Removed LNI 192.168.202.113@tcp [ 1006.388460] Key type .llcrypt unregistered [ 1006.389397] Key type ._llcrypt unregistered [ 1013.314042] Key type ._llcrypt registered [ 1013.314943] Key type .llcrypt registered [ 1013.350690] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 1013.679817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1013.711063] alg: No test for adler32 (adler32-zlib) [ 1014.556912] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 1014.632223] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 1016.207135] Key type lgssc registered [ 1016.547035] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1018.976852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1018.982241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1020.151361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1021.442083] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1024.238156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1024.446831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1025.635886] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1027.367254] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1028.497504] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1030.798779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1031.043743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1032.894213] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1034.603440] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1070.697489] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 13:39:22 (1778348362) [ 1071.246339] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1071.788685] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 13:39:23 (1778348363) [ 1075.679906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1075.684455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1075.691312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1076.192040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1076.193137] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1076.196670] Lustre: Skipped 1 previous similar message [ 1078.868828] Lustre: server umount lustre-MDT0000 complete [ 1080.052205] LustreError: 89634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778348373 with bad export cookie 286380280287080136 [ 1080.056354] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1080.057781] LustreError: 89634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1080.164863] Lustre: server umount lustre-MDT0001 complete [ 1090.620198] Lustre: server umount lustre-OST0000 complete [ 1094.377107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1094.536843] LustreError: 93563:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1094.542160] LustreError: 93563:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1094.582307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1095.884786] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1098.626035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1098.839159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1100.116570] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1101.922647] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1103.252404] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1107.525467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1107.616176] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1107.619728] LustreError: 93551:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1107.632067] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1107.638368] LustreError: 95419:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -98 [ 1107.643712] LustreError: 95419:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 1107.646399] LustreError: 95419:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 1107.676516] Lustre: server umount lustre-OST0000 complete [ 1107.677991] LustreError: 95419:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1110.611739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1110.681561] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1110.683938] LustreError: 93551:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1110.686963] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1110.689505] LustreError: 95881:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -98 [ 1110.691489] LustreError: 95881:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 1110.693178] LustreError: 95881:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 1110.712427] Lustre: server umount lustre-OST0000 complete [ 1110.713590] LustreError: 95881:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1113.835664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1113.905300] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1113.907126] Lustre: Found index 0 for lustre-OST0000, updating log [ 1113.909375] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1113.934670] Lustre: lustre-OST0000: new disk, initializing [ 1113.936533] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1114.029957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1116.045843] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1117.289818] Lustre: Failing over lustre-OST0000 [ 1117.357511] Lustre: server umount lustre-OST0000 complete [ 1119.200944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1119.201066] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1119.201650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1119.201655] Lustre: Skipped 2 previous similar messages [ 1119.209709] Lustre: Skipped 1 previous similar message [ 1124.320612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1124.323225] Lustre: Skipped 2 previous similar messages [ 1124.818878] Lustre: server umount lustre-MDT0000 complete [ 1126.056259] LustreError: 95440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778348419 with bad export cookie 286380280287082866 [ 1126.059542] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1126.061466] LustreError: 95440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1126.173283] Lustre: server umount lustre-MDT0001 complete [ 1129.025458] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 13:40:21 (1778348421) [ 1134.029647] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 1137.475189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1137.615784] LustreError: 98723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1137.643853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1138.885508] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1141.844064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1143.477222] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1144.425890] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1146.587137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1148.630907] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1151.420612] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1151.454478] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1151.487836] Lustre: lustre-OST0001: new disk, initializing [ 1151.489580] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1153.428106] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1154.498266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1154.503357] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1154.530398] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1156.770646] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1163.434844] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1169.891458] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1169.892381] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1169.897040] Lustre: Skipped 2 previous similar messages [ 1169.898757] Lustre: Skipped 1 previous similar message [ 1180.133537] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1180.135704] Lustre: Skipped 8 previous similar messages [ 1184.223154] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1184.293857] Lustre: server umount lustre-MDT0001 complete [ 1185.251659] LustreError: 101442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1185.256902] LustreError: 101442:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1187.480329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1187.514730] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1187.516310] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1187.530027] Lustre: 98712:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1187.775097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1187.777520] Lustre: Skipped 3 previous similar messages [ 1189.209134] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1189.349585] LustreError: 102745:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1189.352263] LustreError: 102745:0:(tgt_mount.c:666:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1189.355222] LustreError: 98715:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1189.382736] LustreError: 102753:0:(obd_config.c:2101:class_config_llog_handler()) MGC192.168.202.113@tcp: cfg command failed: rc = -17 [ 1189.387128] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1189.387128] [ 1192.599870] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 13:41:24 (1778348484) [ 1192.934176] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1198.392821] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 1204.646252] Lustre: 104555:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1212.800408] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1214.331620] Lustre: 105687:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1269.728805] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1269.729260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1269.729390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1269.729398] Lustre: Skipped 1 previous similar message [ 1269.732128] Lustre: Skipped 3 previous similar messages [ 1273.889245] Lustre: server umount lustre-MDT0000 complete [ 1274.849221] LustreError: 100200:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1274.856328] LustreError: 100200:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1275.060798] LustreError: 100989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778348568 with bad export cookie 286380280287083776 [ 1275.062268] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1275.064103] LustreError: 100989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1275.177490] Lustre: server umount lustre-MDT0001 complete [ 1286.148716] Lustre: server umount lustre-OST0000 complete [ 1296.601149] Lustre: server umount lustre-OST0001 complete [ 1298.409599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1301.446631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1304.164825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1307.274097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1313.251604] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 1316.684317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1316.692641] Lustre: lustre-MDT0000: reset Object Index mappings [ 1316.849508] LustreError: 109589:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1316.872822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1318.143840] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1320.946810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1320.955493] Lustre: lustre-MDT0001: reset Object Index mappings [ 1322.497234] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1323.457619] Lustre: 110700:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1325.609158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1326.883405] LustreError: 111053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1326.887653] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 1327.709836] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1330.485154] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1331.690107] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 1331.690996] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 1331.691956] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 1332.490016] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1335.551841] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1336.669821] Lustre: 112551:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1355.472857] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 13:44:07 (1778348647) [ 1388.004233] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1388.005735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1388.008440] Lustre: Skipped 3 previous similar messages [ 1388.012549] Lustre: Skipped 5 previous similar messages [ 1388.014666] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1390.823996] Lustre: server umount lustre-MDT0000 complete [ 1392.077352] LustreError: 109570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778348685 with bad export cookie 286380280287130564 [ 1392.081565] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1392.082651] LustreError: 109570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1393.120172] LustreError: 111873:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1393.124926] LustreError: 111873:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1398.356966] Lustre: server umount lustre-MDT0001 complete [ 1406.148254] Lustre: server umount lustre-OST0000 complete [ 1413.605451] Lustre: server umount lustre-OST0001 complete [ 1415.703095] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 1418.986315] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 1431.343366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1431.451817] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1431.463684] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1431.504853] Lustre: lustre-MDT0000: new disk, initializing [ 1431.537202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1432.927112] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1436.466668] Lustre: server umount lustre-MDT0000 complete [ 1440.569703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1440.716923] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1440.723235] Lustre: 116752:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1440.733385] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1442.417409] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1444.453927] Lustre: server umount lustre-MDT0000 complete [ 1447.636565] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 13:45:39 (1778348739) [ 1448.115525] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1448.708697] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 13:45:40 (1778348740) [ 1454.869470] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 1458.350093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1458.503028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1458.505037] Lustre: Skipped 5 previous similar messages [ 1460.067465] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1463.575656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1463.614337] Lustre: 118768:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1463.638491] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1463.640271] Lustre: Skipped 1 previous similar message [ 1463.678401] Lustre: lustre-MDT0001: new disk, initializing [ 1463.724190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1463.727135] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1465.193348] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1467.738042] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1470.195653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1470.294960] Lustre: lustre-OST0000: new disk, initializing [ 1470.297276] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1470.901105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1470.906783] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1470.944680] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1472.319071] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1476.975093] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1478.600787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1478.604970] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1478.647069] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1479.149908] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1483.802212] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1484.976804] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1650.055911] Lustre: 127529:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 1650.058596] Lustre: 127529:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 1650.062020] Lustre: 127529:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 1650.064100] Lustre: 127529:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 1650.066087] Lustre: 127529:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 1650.067913] Lustre: 127529:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1738.230368] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 13:50:30 (1778349030) [ 1750.496980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1750.502763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1750.510884] Lustre: Skipped 2 previous similar messages [ 1750.514602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1750.518739] Lustre: Skipped 4 previous similar messages [ 1756.287723] Lustre: server umount lustre-MDT0000 complete [ 1757.435100] LustreError: 127555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778349050 with bad export cookie 286380280287182770 [ 1757.438095] LustreError: 127555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1757.439310] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1760.223854] LustreError: 121676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1760.227940] LustreError: 121676:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1763.662748] Lustre: server umount lustre-MDT0001 complete [ 1779.167084] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1779.209344] Lustre: server umount lustre-OST0000 complete [ 1786.818267] Lustre: server umount lustre-OST0001 complete [ 1789.376115] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 1793.276140] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 1805.330880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1805.413781] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1805.423225] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1805.424946] Lustre: Skipped 1 previous similar message [ 1805.460058] Lustre: lustre-MDT0000: new disk, initializing [ 1805.461397] Lustre: Skipped 1 previous similar message [ 1805.482762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1805.484767] Lustre: Skipped 3 previous similar messages [ 1805.489413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1806.745252] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1810.781581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1810.818797] Lustre: 130510:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1810.928121] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1812.179089] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1815.146884] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1816.345917] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1818.581034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1820.216618] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1820.252639] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1820.529695] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1823.581825] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1826.743036] Lustre: lustre-MDD0000: changelog on [ 1827.633685] Lustre: lustre-MDD0001: changelog on [ 2207.656697] Lustre: lustre-MDD0001: changelog off [ 2208.285678] Lustre: 130517:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2208.591238] Lustre: lustre-MDD0000: changelog off [ 2214.370623] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2214.370790] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2214.374018] Lustre: Skipped 5 previous similar messages [ 2214.376506] LustreError: Skipped 1 previous similar message [ 2214.378487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2214.382192] Lustre: Skipped 9 previous similar messages [ 2216.284533] Lustre: server umount lustre-OST0000 complete [ 2223.833988] Lustre: server umount lustre-MDT0000 complete [ 2224.608797] LustreError: 130521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2224.617983] LustreError: 130521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2225.165836] LustreError: 130501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778349518 with bad export cookie 286380280287989863 [ 2225.171699] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2225.172516] LustreError: 130501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2225.311515] Lustre: server umount lustre-MDT0001 complete [ 2229.568380] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 2230.543097] Key type lgssc unregistered [ 2230.671544] LNet: 140647:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2230.673763] LNetError: 140647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2230.685298] LNet: Removed LNI 192.168.202.113@tcp [ 2231.157177] Key type .llcrypt unregistered [ 2231.159300] Key type ._llcrypt unregistered [ 2238.450479] Key type ._llcrypt registered [ 2238.451622] Key type .llcrypt registered [ 2238.491520] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 2244.993458] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 2245.312123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2245.364986] alg: No test for adler32 (adler32-zlib) [ 2246.225185] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 2246.307718] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 2247.887145] Key type lgssc registered [ 2248.272220] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2259.960755] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2259.968310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2261.056399] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2261.068827] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2261.103276] Lustre: lustre-MDT0000: new disk, initializing [ 2261.126830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2261.134297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2262.484862] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2266.712463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2266.768876] Lustre: 143787:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2266.806243] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2266.810073] Lustre: Skipped 1 previous similar message [ 2266.860930] Lustre: lustre-MDT0001: new disk, initializing [ 2266.899535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2266.913557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2266.921141] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2268.402100] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2271.518778] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2272.720681] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2275.217722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2275.314983] Lustre: lustre-OST0000: new disk, initializing [ 2275.316601] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2275.345229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2277.416416] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2279.927319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2279.932203] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2279.963309] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2280.755031] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2282.954930] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_module obdecho/obdecho [ 2283.596983] LustreError: 146405:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 2283.600742] LustreError: 146405:0:(obd_config.c:843:class_setup()) setup ec failed (-22) [ 2284.215337] LustreError: 146501:0:(echo_client.c:1881:echo_md_handler()) Can not resolve the path : rc = -2 [ 2287.584083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2287.587051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2287.592567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2290.145181] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2290.145626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2290.148500] Lustre: Skipped 1 previous similar message [ 2290.151304] Lustre: Skipped 2 previous similar messages [ 2293.528285] Lustre: server umount lustre-MDT0000 complete [ 2294.699256] LustreError: 145531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778349588 with bad export cookie 10040302364682686482 [ 2294.702157] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2294.702636] LustreError: 145531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2294.825933] Lustre: server umount lustre-MDT0001 complete [ 2306.331227] Lustre: server umount lustre-OST0000 complete [ 2313.134902] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 2316.907564] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 2329.048185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2329.148761] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2329.163122] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2329.204806] Lustre: lustre-MDT0000: new disk, initializing [ 2329.231546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2329.239163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2330.718603] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2334.996691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2335.034803] Lustre: 150172:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2335.055539] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2335.056938] Lustre: Skipped 1 previous similar message [ 2335.093982] Lustre: lustre-MDT0001: new disk, initializing [ 2335.124431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2335.135190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2335.138158] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2336.431979] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2339.648652] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2340.942710] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2343.452205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2343.557234] Lustre: lustre-OST0000: new disk, initializing [ 2343.559543] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2343.584174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2343.974521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2343.977541] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2344.021164] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2345.472181] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2348.542175] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2349.916380] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2349.987692] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2351.253782] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2351.320685] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2354.145032] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2354.145812] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2354.148399] LustreError: Skipped 1 previous similar message [ 2354.149588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2354.153805] Lustre: Skipped 1 previous similar message [ 2358.026731] Lustre: server umount lustre-OST0000 complete [ 2360.800196] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2360.803886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2360.810855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2360.812991] Lustre: Skipped 1 previous similar message [ 2365.475138] Lustre: server umount lustre-MDT0000 complete [ 2366.678628] LustreError: 151910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778349660 with bad export cookie 10040302364682687441 [ 2366.682504] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2366.796630] Lustre: server umount lustre-MDT0001 complete [ 2367.957625] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 14:01:00 (1778349660) [ 2368.519847] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 2369.083521] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 14:01:01 (1778349661) [ 2371.680186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2371.896227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2373.279771] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2376.204879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2377.633412] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2379.350159] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2380.563033] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2382.962698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2384.874396] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2386.675313] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2407.391753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2407.393727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2407.396695] Lustre: Skipped 1 previous similar message [ 2407.398249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2407.399725] Lustre: Skipped 2 previous similar messages [ 2410.460269] Lustre: server umount lustre-MDT0000 complete [ 2411.713152] LustreError: 154169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778349705 with bad export cookie 10040302364682688309 [ 2411.717041] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2411.717091] LustreError: 154169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2412.546194] LustreError: 154184:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2412.559510] LustreError: 154184:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2416.031667] LustreError: 154183:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2416.031924] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2416.035666] LustreError: 154183:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2416.039387] Lustre: Skipped 5 previous similar messages [ 2417.665175] LustreError: 154182:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2421.215801] LustreError: 154212:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2426.335088] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2426.419153] Lustre: server umount lustre-MDT0001 complete [ 2428.469884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2432.510659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2432.584699] LustreError: 157714:0:(llog.c:1646:llog_backup()) MGC192.168.202.113@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2432.587299] Lustre: 157714:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.202.113@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2451.935926] LustreError: 157719:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2451.939964] LustreError: 157719:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2462.175760] LustreError: 157737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2462.182571] LustreError: 157737:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 2478.560112] LustreError: 157720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2478.568125] LustreError: 157720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 2493.919284] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2493.922494] Lustre: 157714:0:(mgc_request.c:1917:mgc_process_log()) MGC192.168.202.113@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2512.352726] LustreError: 157720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2512.363310] LustreError: 157720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 20 previous similar messages [ 2534.441178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2534.443251] Lustre: Skipped 2 previous similar messages [ 2535.825938] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2536.483434] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2539.049582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2556.898851] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8b5651ce0c6d3080 [ 2556.902442] Lustre: MGC192.168.202.113@tcp: Connection restored to 0@lo (at 0@lo) [ 2558.152602] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2560.103724] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2560.634123] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2561.847213] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2562.020943] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2565.499434] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2572.114849] Lustre: server umount lustre-MDT0000 complete [ 2572.256679] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2572.258096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2572.262332] Lustre: Skipped 4 previous similar messages [ 2573.310174] LustreError: 154213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778349866 with bad export cookie 10040302364682694784 [ 2573.313732] LustreError: 154213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2573.314486] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2573.434283] Lustre: server umount lustre-MDT0001 complete [ 2585.049403] Lustre: server umount lustre-OST0000 complete [ 2592.873171] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 2596.410024] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 2609.352882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2609.482838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2609.494344] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2609.539793] Lustre: lustre-MDT0000: new disk, initializing [ 2609.564196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2609.566282] Lustre: Skipped 1 previous similar message [ 2609.571196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2611.028802] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2615.504564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2615.542777] Lustre: 162532:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2615.641891] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2616.906825] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2620.201305] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2621.340369] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2623.587747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2624.439085] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2624.472540] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2625.458763] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2628.853125] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2630.360546] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2630.435442] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2631.933889] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2632.008429] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2634.724081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2634.724183] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2634.727934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2634.727944] Lustre: Skipped 4 previous similar messages [ 2634.729303] LustreError: Skipped 1 previous similar message [ 2634.737828] Lustre: Skipped 2 previous similar messages [ 2638.986746] Lustre: server umount lustre-OST0000 complete [ 2646.744751] Lustre: server umount lustre-MDT0000 complete [ 2648.201953] LustreError: 162524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778349941 with bad export cookie 10040302364682711584 [ 2648.205211] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2648.206381] LustreError: 162524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2648.314733] Lustre: server umount lustre-MDT0001 complete [ 2651.311221] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 14:05:43 (1778349943) [ 2653.447423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2653.587526] LustreError: 166533:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2653.592312] LustreError: 166533:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [ 2654.914864] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2657.689141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2659.094830] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2660.866593] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2662.133964] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2664.305702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2666.294631] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2668.067691] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2682.594553] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 14:06:14 (1778349974) [ 2683.872632] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2683.872693] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2683.874416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2683.874420] Lustre: Skipped 4 previous similar messages [ 2683.874728] LustreError: Skipped 1 previous similar message [ 2683.877870] Lustre: Skipped 3 previous similar messages [ 2685.643898] Lustre: server umount lustre-OST0000 complete [ 2693.071061] Lustre: server umount lustre-MDT0000 complete [ 2694.287935] LustreError: 166514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778349987 with bad export cookie 10040302364682712452 [ 2694.291076] LustreError: 166514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2694.291585] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2694.390933] Lustre: server umount lustre-MDT0001 complete [ 2697.834211] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 2698.777532] Key type lgssc unregistered [ 2698.905299] LNet: 169926:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2698.908390] LNetError: 169926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2698.918404] LNet: Removed LNI 192.168.202.113@tcp [ 2699.236963] Key type .llcrypt unregistered [ 2699.238361] Key type ._llcrypt unregistered [ 2706.854964] Key type ._llcrypt registered [ 2706.855836] Key type .llcrypt registered [ 2706.892951] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 2707.306658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2707.355587] alg: No test for adler32 (adler32-zlib) [ 2708.218731] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 2708.300355] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 2709.879176] Key type lgssc registered [ 2710.219792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2712.752895] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2712.763497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2728.223388] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2733.343144] LustreError: 171130:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.113@tcp: failed processing log, type 4: rc = -110 [ 2738.463335] LustreError: 171130:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2738.469310] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 2738.472451] LustreError: MGC192.168.202.113@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 2738.477662] LustreError: 171130:0:(tgt_mount.c:1819:server_start_targets()) failed to start server lustre-OST0000: -22 [ 2738.482451] LustreError: 171130:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -22 [ 2738.484866] LustreError: 171130:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 2738.504423] Lustre: server umount lustre-OST0000 complete [ 2738.505637] LustreError: 171130:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 2741.518328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2742.665718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2743.855180] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2746.622924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2746.741915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2748.103522] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2750.250489] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2751.636880] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2754.148826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2754.223020] LustreError: 173214:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2754.225894] Lustre: 173214:0:(mgc_request_server.c:743:mgc_llog_local_copy()) MGC192.168.202.113@tcp: invalid local config log lustre-OST0000: rc = -22 [ 2754.278813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2756.442942] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2758.371717] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2762.207812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2762.210570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2762.215717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2762.722916] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2762.730267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2762.734290] Lustre: Skipped 1 previous similar message [ 2765.407738] Lustre: server umount lustre-MDT0000 complete [ 2766.729063] LustreError: 171609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778350060 with bad export cookie 13592474908612037890 [ 2766.731797] LustreError: 171609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2766.732081] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2766.837340] Lustre: server umount lustre-MDT0001 complete [ 2778.337143] Lustre: server umount lustre-OST0000 complete [ 2781.324492] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 2784.427725] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 2797.546198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2797.643757] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2797.655327] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2797.694640] Lustre: lustre-MDT0000: new disk, initializing [ 2797.720512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2797.725851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2799.342142] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2803.838652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2803.885291] Lustre: 176588:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2803.911880] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2803.914470] Lustre: Skipped 1 previous similar message [ 2803.953517] Lustre: lustre-MDT0001: new disk, initializing [ 2803.987160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2804.000361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2804.003181] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2805.565538] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2808.704095] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2809.790571] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2812.170494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2812.316431] Lustre: lustre-OST0000: new disk, initializing [ 2812.319953] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2812.365080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2814.193695] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2817.683088] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2819.064645] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2820.115218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2820.121677] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2820.179983] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2821.266888] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2822.779370] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2822.887107] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2825.192199] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2825.193925] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2825.196854] LustreError: Skipped 1 previous similar message [ 2825.204800] Lustre: Skipped 1 previous similar message [ 2825.208611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2825.213885] Lustre: Skipped 1 previous similar message [ 2829.683641] Lustre: server umount lustre-OST0000 complete [ 2834.912463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2834.916828] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2834.925055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2837.604819] Lustre: server umount lustre-MDT0000 complete [ 2838.833247] LustreError: 176581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778350132 with bad export cookie 13592474908612038590 [ 2838.835721] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2838.837432] LustreError: 176581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2838.965739] Lustre: server umount lustre-MDT0001 complete [ 2842.228706] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 14:08:54 (1778350134) [ 2848.935064] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 2849.872467] Key type lgssc unregistered [ 2849.987345] LNet: 181119:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2849.989282] LNetError: 181119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2849.997268] LNet: Removed LNI 192.168.202.113@tcp [ 2850.278152] Key type .llcrypt unregistered [ 2850.279394] Key type ._llcrypt unregistered [ 2857.133394] Key type ._llcrypt registered [ 2857.135140] Key type .llcrypt registered [ 2857.172685] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 2857.463491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2857.477800] alg: No test for adler32 (adler32-zlib) [ 2858.328324] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 2858.405232] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 2859.983263] Key type lgssc registered [ 2860.377453] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2862.935484] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2862.941137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2864.100913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2865.333650] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2869.073086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2869.188409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2870.458407] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2872.199156] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2873.427378] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2876.073065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2876.144459] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2876.146117] LustreError: 183934:0:(llog.c:1687:llog_backup()) MGC192.168.202.113@tcp: failed to backup log lustre-OST0000: rc = -28 [ 2876.149873] Lustre: 183934:0:(mgc_request_server.c:740:mgc_llog_local_copy()) MGC192.168.202.113@tcp: can't backup local config lustre-OST0000: rc = -28 [ 2876.190737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2878.029598] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2879.806865] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2884.577755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2884.578480] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2884.579666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2884.586008] Lustre: Skipped 2 previous similar messages [ 2886.738844] Lustre: server umount lustre-MDT0000 complete [ 2887.908705] LustreError: 182279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778350181 with bad export cookie 9957746990910749764 [ 2887.914253] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2887.915679] LustreError: 182279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2888.030462] Lustre: server umount lustre-MDT0001 complete [ 2899.795834] Lustre: server umount lustre-OST0000 complete [ 2902.461623] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 2905.421870] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 2916.178360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2916.265531] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2916.274740] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2916.308662] Lustre: lustre-MDT0000: new disk, initializing [ 2916.329993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2916.335073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2917.523175] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2921.482204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2921.518986] Lustre: 187310:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2921.535396] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2921.536662] Lustre: Skipped 1 previous similar message [ 2921.537912] LustreError: 188076:0:(llog.c:1687:llog_backup()) MGC192.168.202.113@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 2921.540349] LustreError: 188076:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 2921.542202] Lustre: 188076:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.202.113@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 2921.548756] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2921.550357] Lustre: Skipped 1 previous similar message [ 2921.585988] Lustre: lustre-MDT0001: new disk, initializing [ 2921.617598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2921.628548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2921.631746] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2922.906114] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2925.846428] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2927.008597] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2929.223455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2929.318474] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2929.320733] Lustre: Skipped 1 previous similar message [ 2929.323254] LustreError: 189029:0:(llog.c:1687:llog_backup()) MGC192.168.202.113@tcp: failed to backup log lustre-OST0000: rc = -28 [ 2929.328213] LustreError: 189029:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 2929.331989] Lustre: 189029:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.202.113@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 2929.337371] Lustre: 189029:0:(mgc_request_server.c:763:mgc_llog_local_copy()) Skipped 1 previous similar message [ 2929.347679] Lustre: lustre-OST0000: new disk, initializing [ 2929.350414] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2931.095851] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2931.965132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2931.968724] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2932.006784] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2934.249980] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2935.618374] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2935.682842] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2936.946073] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2937.016704] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2942.434410] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2942.434500] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2942.435494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2942.435498] Lustre: Skipped 2 previous similar messages [ 2942.438687] Lustre: Skipped 1 previous similar message [ 2942.440862] LustreError: Skipped 1 previous similar message [ 2947.554709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2947.556423] Lustre: Skipped 2 previous similar messages [ 2952.159111] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2952.212089] Lustre: server umount lustre-OST0000 complete [ 2957.792433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2957.792818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2957.800681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2957.803408] Lustre: Skipped 1 previous similar message [ 2959.515694] Lustre: server umount lustre-MDT0000 complete [ 2960.656041] LustreError: 187302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778350254 with bad export cookie 9957746990910750464 [ 2960.658645] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2960.659717] LustreError: 187302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2960.768304] Lustre: server umount lustre-MDT0001 complete [ 2963.356582] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 14:10:55 (1778350255) [ 2968.349660] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 2971.424373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2971.564489] LustreError: 192009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2971.570367] LustreError: 192009:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2971.577622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2971.579408] Lustre: Skipped 1 previous similar message [ 2972.748455] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2975.354028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2976.649309] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2977.568682] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2979.570207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2981.441264] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2984.110371] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2984.169299] Lustre: lustre-OST0001: new disk, initializing [ 2984.171247] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2984.475852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2984.480441] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2984.512603] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2986.048843] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2991.517596] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2999.205474] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3004.833239] Lustre: DEBUG MARKER: ADD OST3 [ 3006.403183] loop: module loaded [ 3006.421148] loop0: detected capacity change from 0 to 204800000 [ 3006.529886] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3006.533742] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3006.536604] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3008.485746] loop0: detected capacity change from 0 to 204800000 [ 3010.318166] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3010.377635] Lustre: lustre-OST0003: new disk, initializing [ 3010.380297] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 3010.417818] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 3010.420371] Lustre: Skipped 3 previous similar messages [ 3011.474879] Lustre: DEBUG MARKER: STOP OST3 [ 3011.595532] Lustre: *** cfs_fail_loc=2109, val=2*** [ 3011.624311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 3011.627023] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 3011.636803] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 3012.164978] Lustre: Failing over lustre-OST0003 [ 3012.211720] Lustre: server umount lustre-OST0003 complete [ 3012.407971] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3013.599092] LustreError: 196214:0:(osp_precreate.c:1280:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 3014.472027] LustreError: 193459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3016.467193] Lustre: DEBUG MARKER: START OST3 again [ 3016.513821] LustreError: 193460:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3016.677468] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3016.680943] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3016.686571] Lustre: Skipped 1 previous similar message [ 3018.690515] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3018.790508] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 3020.492065] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3020.615265] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3020.660022] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3020.660064] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3034.147775] Lustre: Failing over lustre-OST0003 [ 3034.185779] Lustre: server umount lustre-OST0003 complete [ 3036.129051] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3036.129959] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3036.133379] LustreError: Skipped 1 previous similar message [ 3036.134299] LustreError: 193460:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3036.136705] Lustre: Skipped 2 previous similar messages [ 3036.142808] LustreError: 193460:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3044.322642] LustreError: 193857:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3044.326627] LustreError: 193857:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3047.391777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3047.393679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3047.397162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3054.564227] LustreError: 193857:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3054.569281] LustreError: 193857:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3059.680566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3059.682267] Lustre: Skipped 9 previous similar messages [ 3061.215087] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3061.282150] Lustre: server umount lustre-MDT0000 complete [ 3062.459188] LustreError: 191990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778350356 with bad export cookie 9957746990910751332 [ 3062.463154] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3062.464790] LustreError: 191990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3064.800838] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3064.804573] Lustre: Skipped 4 previous similar messages [ 3068.662674] Lustre: server umount lustre-MDT0001 complete [ 3070.100921] Lustre: server umount lustre-OST0000 complete [ 3071.518592] Lustre: server umount lustre-OST0001 complete [ 3073.768021] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 3076.578687] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 3088.684324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3088.773389] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3088.783815] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3088.821501] Lustre: lustre-MDT0000: new disk, initializing [ 3088.845092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3088.846954] Lustre: Skipped 1 previous similar message [ 3088.852484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3090.092401] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3094.491444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3094.529061] Lustre: 201328:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3094.626406] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3096.058399] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3099.343658] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3100.695685] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3103.030942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3105.022718] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3105.082850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3105.085132] Lustre: Skipped 1 previous similar message [ 3105.086935] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3105.122106] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3108.166717] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3109.514951] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3109.577221] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3110.877028] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3110.939777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3115.490686] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3115.491340] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3115.492033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3115.492037] Lustre: Skipped 4 previous similar messages [ 3115.495242] Lustre: Skipped 1 previous similar message [ 3117.708083] Lustre: server umount lustre-OST0000 complete [ 3133.407108] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3133.463250] Lustre: server umount lustre-MDT0000 complete [ 3134.696247] LustreError: 201321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778350428 with bad export cookie 9957746990910753586 [ 3134.699541] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3134.701146] LustreError: 201321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3134.822427] Lustre: server umount lustre-MDT0001 complete [ 3136.077590] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 14:13:48 (1778350428) [ 3142.027427] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 3145.136726] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 3156.025940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3156.110260] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3156.120264] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3156.121967] Lustre: Skipped 3 previous similar messages [ 3156.157627] Lustre: lustre-MDT0000: new disk, initializing [ 3156.159744] Lustre: Skipped 2 previous similar messages [ 3156.185102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3156.186750] Lustre: Skipped 2 previous similar messages [ 3156.191804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3157.430139] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3161.482989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3161.518293] Lustre: 207500:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3161.610686] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3162.817333] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3165.840841] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3167.059236] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3169.282244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3170.777074] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3171.211348] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3174.244941] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3175.533806] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3175.610797] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3176.869887] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3176.938445] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3181.025955] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3181.026296] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3181.030195] Lustre: Skipped 2 previous similar messages [ 3181.032233] LustreError: Skipped 3 previous similar messages [ 3181.034054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3181.037102] Lustre: Skipped 8 previous similar messages [ 3183.694489] Lustre: server umount lustre-OST0000 complete [ 3191.134475] Lustre: server umount lustre-MDT0000 complete [ 3191.264381] LustreError: 208284:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3191.269040] LustreError: 208284:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 3192.359256] LustreError: 207491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778350485 with bad export cookie 9957746990910754468 [ 3192.362457] LustreError: 207491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3192.362873] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3192.486906] Lustre: server umount lustre-MDT0001 complete [ 3195.322156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3196.798198] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3199.636494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3201.022216] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3202.773688] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3203.913661] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3206.048735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3208.026211] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3209.785642] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3222.417714] Lustre: server umount lustre-OST0000 complete [ 3229.841674] Lustre: server umount lustre-MDT0000 complete [ 3230.687928] LustreError: 212019:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3230.691930] LustreError: 212019:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3231.055307] LustreError: 212924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778350524 with bad export cookie 9957746990910755350 [ 3231.058705] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3231.061203] LustreError: 212924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3231.176571] Lustre: server umount lustre-MDT0001 complete [ 3341.328289] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 3342.275676] Key type lgssc unregistered [ 3342.398351] LNet: 214615:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3342.400740] LNetError: 214615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3342.409304] LNet: Removed LNI 192.168.202.113@tcp [ 3342.709109] Key type .llcrypt unregistered [ 3342.710577] Key type ._llcrypt unregistered [ 3350.643256] Key type ._llcrypt registered [ 3350.645804] Key type .llcrypt registered [ 3350.680904] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 3356.678854] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 3357.136422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3357.150376] alg: No test for adler32 (adler32-zlib) [ 3358.007508] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 3358.092643] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 3359.671173] Key type lgssc registered [ 3360.037361] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3370.511758] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3370.516739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3371.609165] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3371.620143] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3371.660481] Lustre: lustre-MDT0000: new disk, initializing [ 3371.687532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3371.693816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3372.935797] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3377.324327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3377.389650] Lustre: 217905:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3377.429811] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3377.435329] Lustre: Skipped 1 previous similar message [ 3377.484832] Lustre: lustre-MDT0001: new disk, initializing [ 3377.532934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3377.550523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3377.557291] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3379.203960] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3382.368539] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3383.794696] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3386.306639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3386.418810] Lustre: lustre-OST0000: new disk, initializing [ 3386.420614] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3386.450698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3388.263447] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3391.218141] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3392.580920] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3395.067803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3395.071685] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3395.111699] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3395.739500] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3397.043552] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3397.124402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3400.160860] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3400.161933] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3400.163643] LustreError: Skipped 1 previous similar message [ 3400.167348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3400.169454] Lustre: Skipped 1 previous similar message [ 3403.863499] Lustre: server umount lustre-OST0000 complete [ 3405.280406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3405.281038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3405.283582] Lustre: Skipped 2 previous similar messages [ 3410.399968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3410.401415] Lustre: Skipped 2 previous similar messages [ 3411.221792] Lustre: server umount lustre-MDT0000 complete [ 3412.380121] LustreError: 218682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778350705 with bad export cookie 9855000427549476936 [ 3412.382872] LustreError: 218682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3412.383328] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3412.497323] Lustre: server umount lustre-MDT0001 complete [ 3417.927839] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 3421.106173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3421.253816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3422.362732] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3424.968644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3426.276513] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3427.175110] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3429.222245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3429.337443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3429.339117] Lustre: Skipped 1 previous similar message [ 3431.012737] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3433.605141] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3433.659394] Lustre: lustre-OST0001: new disk, initializing [ 3433.661565] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3434.365734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3434.368613] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3434.397845] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3435.382385] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3440.566283] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3445.079071] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3453.506842] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 14:19:05 (1778350745) [ 3455.976399] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3455.977062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3455.980294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3455.981512] Lustre: Skipped 2 previous similar messages [ 3455.984467] Lustre: Skipped 3 previous similar messages [ 3460.322593] Lustre: server umount lustre-MDT0000 complete [ 3461.091585] LustreError: 222435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3461.095903] LustreError: 222435:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3461.534519] LustreError: 222416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778350755 with bad export cookie 9855000427549477804 [ 3461.537605] LustreError: 222416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3461.540031] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3461.652274] Lustre: server umount lustre-MDT0001 complete [ 3473.407931] Lustre: server umount lustre-OST0000 complete [ 3484.825384] Lustre: server umount lustre-OST0001 complete [ 3486.800587] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 3489.696276] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 3500.868193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3500.955289] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3500.964900] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3500.999852] Lustre: lustre-MDT0000: new disk, initializing [ 3501.023816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3501.025560] Lustre: Skipped 1 previous similar message [ 3501.030577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3502.204414] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3506.067231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3506.103810] Lustre: 228581:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3506.202448] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3507.382555] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3510.317421] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3511.437959] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3513.581510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3513.685473] Lustre: lustre-OST0000: new disk, initializing [ 3513.686694] Lustre: Skipped 1 previous similar message [ 3513.688502] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3513.691742] Lustre: Skipped 2 previous similar messages [ 3515.514373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3515.516430] Lustre: Skipped 1 previous similar message [ 3515.518520] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3515.553702] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3515.581541] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3518.529754] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3519.852717] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3519.925303] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3521.188100] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3521.265896] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3526.115919] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3526.118038] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3526.120915] Lustre: Skipped 2 previous similar messages [ 3526.121644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3526.124276] Lustre: Skipped 1 previous similar message [ 3536.351183] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3536.356268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3536.359179] Lustre: Skipped 3 previous similar messages [ 3536.415267] Lustre: server umount lustre-OST0000 complete [ 3541.473462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3541.476554] Lustre: Skipped 1 previous similar message [ 3543.771807] Lustre: server umount lustre-MDT0000 complete [ 3544.953328] LustreError: 230319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778350838 with bad export cookie 9855000427549479904 [ 3544.956927] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3544.958036] LustreError: 230319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3545.083932] Lustre: server umount lustre-MDT0001 complete [ 3547.802100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3547.984399] LustreError: 232389:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3548.012913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3548.015774] Lustre: Skipped 2 previous similar messages [ 3549.354527] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3552.168748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3553.620348] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3555.357968] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3556.475271] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3558.666040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3560.747322] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3562.489229] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3710.431992] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3710.433970] LustreError: Skipped 1 previous similar message [ 3710.435146] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3710.439078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3710.441449] Lustre: Skipped 2 previous similar messages [ 3722.207081] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3722.252411] Lustre: server umount lustre-OST0000 complete [ 3726.303856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3726.305756] LustreError: Skipped 1 previous similar message [ 3726.816649] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3726.822261] Lustre: Skipped 2 previous similar messages [ 3729.552641] Lustre: server umount lustre-MDT0000 complete [ 3730.687227] LustreError: 232369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778351024 with bad export cookie 9855000427549480786 [ 3730.690053] LustreError: 232369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3730.690454] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3730.808304] Lustre: server umount lustre-MDT0001 complete [ 3824.438089] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 3825.469185] Key type lgssc unregistered [ 3825.598462] LNet: 235640:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3825.602220] LNetError: 235640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3825.611401] LNet: Removed LNI 192.168.202.113@tcp [ 3825.919151] Key type .llcrypt unregistered [ 3825.920325] Key type ._llcrypt unregistered [ 3833.683199] Key type ._llcrypt registered [ 3833.684243] Key type .llcrypt registered [ 3833.716901] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 3839.925634] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 3840.290854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3840.373611] alg: No test for adler32 (adler32-zlib) [ 3841.230656] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 3841.322515] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 3842.903157] Key type lgssc registered [ 3843.242715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3857.078546] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 3860.568048] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3860.573309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3861.677119] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3861.689172] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3861.736911] Lustre: lustre-MDT0000: new disk, initializing [ 3861.763245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3861.769597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3863.030435] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3867.113036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3867.153024] Lustre: 239631:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3867.173558] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3867.175349] Lustre: Skipped 1 previous similar message [ 3867.215853] Lustre: lustre-MDT0001: new disk, initializing [ 3867.246987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3867.257578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3867.260529] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3868.530843] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3870.819933] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3872.931053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3873.039075] Lustre: lustre-OST0000: new disk, initializing [ 3873.041156] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3873.067646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3873.911859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3873.915069] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3873.968791] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3874.818583] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3878.897641] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3878.959122] Lustre: lustre-OST0001: new disk, initializing [ 3878.961020] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3878.993532] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3880.831025] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3883.519349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3883.524521] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3883.556959] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3886.250699] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3889.844608] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3898.336214] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3898.338388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3898.342357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3898.853385] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3898.856508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3898.857523] Lustre: Skipped 2 previous similar messages [ 3898.859185] Lustre: Skipped 1 previous similar message [ 3903.833425] Lustre: server umount lustre-MDT0000 complete [ 3905.581393] LustreError: 240408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778351199 with bad export cookie 17551863227755750832 [ 3905.584047] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3905.584374] LustreError: 240408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3905.704857] Lustre: server umount lustre-MDT0001 complete [ 3917.316416] Lustre: server umount lustre-OST0000 complete [ 3927.645415] Lustre: server umount lustre-OST0001 complete [ 3932.025169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3934.995879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3937.726770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3940.641818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3946.554488] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 3950.047696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3950.053906] Lustre: lustre-MDT0000: reset Object Index mappings [ 3950.202149] LustreError: 247049:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3950.207633] LustreError: 247049:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3950.223215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3951.511836] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3954.188020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3954.193577] Lustre: lustre-MDT0001: reset Object Index mappings [ 3955.588855] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3956.498047] Lustre: 248158:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3958.614662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3958.743434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3958.745812] Lustre: Skipped 1 previous similar message [ 3959.783431] LustreError: 248512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3959.786815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3960.579950] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3963.322604] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3964.464683] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 3965.275301] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3968.272208] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3969.379265] Lustre: 250000:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3979.746835] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3979.749783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3979.752942] Lustre: Skipped 2 previous similar messages [ 3979.755674] Lustre: Skipped 2 previous similar messages [ 3981.660167] Lustre: server umount lustre-MDT0000 complete [ 3982.840650] LustreError: 247030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778351276 with bad export cookie 17551863227755757552 [ 3982.843774] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3982.846184] LustreError: 247030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3982.973594] Lustre: server umount lustre-MDT0001 complete [ 3994.048992] Lustre: server umount lustre-OST0000 complete [ 4005.530644] Lustre: server umount lustre-OST0001 complete [ 4011.616100] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 14:28:23 (1778351303) [ 4016.574337] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 4019.424569] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 4031.292471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4031.392043] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4031.405555] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4031.443716] Lustre: lustre-MDT0000: new disk, initializing [ 4031.470031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4031.471734] Lustre: Skipped 1 previous similar message [ 4031.476722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4032.813384] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4037.012793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4037.055334] Lustre: 254216:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4037.159390] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4038.565850] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4041.652479] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4042.806817] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4045.115290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4045.244180] Lustre: lustre-OST0000: new disk, initializing [ 4045.245785] Lustre: Skipped 1 previous similar message [ 4045.247695] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4045.249389] Lustre: Skipped 2 previous similar messages [ 4046.329676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4046.332195] Lustre: Skipped 1 previous similar message [ 4046.334044] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4046.361414] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4047.126840] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4050.222982] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4051.576420] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4051.649284] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4052.975100] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4053.062382] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4056.548461] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4056.548729] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4056.550499] LustreError: Skipped 1 previous similar message [ 4056.551134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4056.553920] Lustre: Skipped 2 previous similar messages [ 4056.555069] Lustre: Skipped 2 previous similar messages [ 4059.798850] Lustre: server umount lustre-OST0000 complete [ 4061.665214] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4061.665819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4061.668466] Lustre: Skipped 1 previous similar message [ 4061.671131] Lustre: Skipped 2 previous similar messages [ 4067.160193] Lustre: server umount lustre-MDT0000 complete [ 4068.354378] LustreError: 254208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778351361 with bad export cookie 17551863227755759645 [ 4068.357303] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4068.357750] LustreError: 254208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4068.476030] Lustre: server umount lustre-MDT0001 complete [ 4074.054791] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 4077.455274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4077.590648] LustreError: 258721:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4077.604822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4077.606893] Lustre: Skipped 2 previous similar messages [ 4078.845402] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4081.794553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4083.216828] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4084.259210] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4086.539821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4088.544984] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4091.451571] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4091.513119] Lustre: lustre-OST0001: new disk, initializing [ 4091.515946] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4092.871968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4092.877291] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4092.913816] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4093.480907] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4097.926992] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4099.012396] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4105.736097] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4105.744883] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4107.744753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4107.750743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4107.760623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4107.764287] Lustre: Skipped 2 previous similar messages [ 4113.690395] Lustre: server umount lustre-MDT0000 complete [ 4115.259154] LustreError: 258704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778351408 with bad export cookie 17551863227755760527 [ 4115.267109] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4115.269154] LustreError: 258704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4115.400593] Lustre: server umount lustre-MDT0001 complete [ 4126.724735] Lustre: server umount lustre-OST0000 complete [ 4138.145890] Lustre: server umount lustre-OST0001 complete [ 4143.145787] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 4146.388461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4146.525076] LustreError: 263996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4146.543589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4146.545395] Lustre: Skipped 3 previous similar messages [ 4147.729438] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4150.317507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4151.613899] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4152.513485] Lustre: 265102:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4154.592246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4156.564194] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4156.773601] LustreError: 265453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4156.781978] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 4159.680104] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4160.821787] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 4161.826124] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4165.024874] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4166.191689] Lustre: 266948:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4172.937753] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 4172.942406] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 4175.996508] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 14:31:08 (1778351468) [ 4191.715491] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4191.718740] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4191.718880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4191.718888] Lustre: Skipped 8 previous similar messages [ 4191.719096] Lustre: Skipped 5 previous similar messages [ 4193.630496] Lustre: server umount lustre-MDT0000 complete [ 4194.813549] LustreError: 264721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778351488 with bad export cookie 17551863227755766820 [ 4194.818451] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4196.832566] LustreError: 263992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4196.836549] LustreError: 263992:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4209.119169] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4209.211152] Lustre: server umount lustre-MDT0001 complete [ 4216.443271] Lustre: server umount lustre-OST0000 complete [ 4223.636568] Lustre: server umount lustre-OST0001 complete [ 4225.472817] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 4228.156809] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 4241.923086] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 4245.304475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4245.405631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4245.415252] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4245.448908] Lustre: lustre-MDT0000: new disk, initializing [ 4245.476056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4246.694229] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4250.676905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4250.713871] Lustre: 271128:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4250.819817] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4252.047511] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4256.349166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4257.304623] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4258.218654] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4262.172307] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4264.062535] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4264.106849] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4268.348924] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4278.302283] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 14:32:50 (1778351570) [ 4279.778386] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4279.779339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4279.783438] Lustre: Skipped 3 previous similar messages [ 4279.784843] Lustre: Skipped 9 previous similar messages [ 4285.087120] Lustre: server umount lustre-MDT0000 complete [ 4286.240447] LustreError: 271905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778351579 with bad export cookie 17551863227755774688 [ 4286.244059] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4286.244285] LustreError: 271905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4286.347361] Lustre: server umount lustre-MDT0001 complete [ 4298.245626] Lustre: server umount lustre-OST0000 complete [ 4308.636233] Lustre: server umount lustre-OST0001 complete [ 4313.885452] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 4316.537726] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 4327.497097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4327.597791] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4327.600503] Lustre: Skipped 2 previous similar messages [ 4327.609209] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4327.610645] Lustre: Skipped 4 previous similar messages [ 4327.649485] Lustre: lustre-MDT0000: new disk, initializing [ 4327.650703] Lustre: Skipped 3 previous similar messages [ 4327.674111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4327.675741] Lustre: Skipped 7 previous similar messages [ 4327.680481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4327.682465] Lustre: Skipped 3 previous similar messages [ 4328.900871] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4332.715859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4332.752718] Lustre: 278161:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4332.851586] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4332.854460] Lustre: Skipped 2 previous similar messages [ 4333.999697] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4336.883025] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4338.017394] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4340.036584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4341.872117] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4341.901894] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4344.826528] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4345.487768] Lustre: Mounted lustre-client [ 4345.764973] LustreError: 280658:0:(lov_obd.c:786:lov_cleanup()) lustre-clilov-ffff9972454df800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 4345.793714] Lustre: Unmounted lustre-client [ 4350.945911] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4350.947549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4350.949037] Lustre: Skipped 3 previous similar messages [ 4350.953519] Lustre: Skipped 8 previous similar messages [ 4354.778325] Lustre: server umount lustre-OST0000 complete [ 4362.072322] Lustre: server umount lustre-MDT0000 complete [ 4363.215272] LustreError: 278153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778351656 with bad export cookie 17551863227755776214 [ 4363.218470] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4363.219804] LustreError: 278153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4363.325091] Lustre: server umount lustre-MDT0001 complete [ 4366.509493] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 4367.406105] Key type lgssc unregistered [ 4408.479389] LNet: 281938:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4408.482652] LNetError: 281938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4474.021398] LNet: Removed LNI 192.168.202.113@tcp [ 4474.303117] Key type .llcrypt unregistered [ 4474.304737] Key type ._llcrypt unregistered [ 4479.552963] Key type ._llcrypt registered [ 4479.553958] Key type .llcrypt registered [ 4479.585314] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 4485.359291] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 4485.794684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4485.828672] alg: No test for adler32 (adler32-zlib) [ 4486.688103] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 4486.780865] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 4488.359129] Key type lgssc registered [ 4488.675032] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4497.518330] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 14:36:29 (1778351789) [ 4499.460780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4499.465254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4500.554414] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4500.565441] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4500.600555] Lustre: lustre-MDT0000: new disk, initializing [ 4500.623579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4500.628974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4501.756504] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4505.535729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4505.574880] Lustre: 285098:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4505.593920] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4505.595474] Lustre: Skipped 1 previous similar message [ 4505.631713] Lustre: lustre-MDT0001: new disk, initializing [ 4505.663959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4505.675356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4505.679269] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4506.797300] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4509.611573] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4510.628420] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4512.636317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4512.740342] Lustre: lustre-OST0000: new disk, initializing [ 4512.741924] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4512.766262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4514.390066] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4517.209944] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4519.413597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4519.416295] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4519.443226] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4529.633276] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4529.633367] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4529.637958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4529.639355] LustreError: Skipped 1 previous similar message [ 4532.110887] Lustre: server umount lustre-OST0000 complete [ 4534.752749] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4534.753329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4534.755586] Lustre: Skipped 2 previous similar messages [ 4534.757992] Lustre: Skipped 2 previous similar messages [ 4539.350605] Lustre: server umount lustre-MDT0000 complete [ 4540.455055] LustreError: 285088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778351834 with bad export cookie 17935839374567848077 [ 4540.457384] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4540.458156] LustreError: 285088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4540.571037] Lustre: server umount lustre-MDT0001 complete [ 4543.619822] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 4544.483179] Key type lgssc unregistered [ 4544.594373] LNet: 288469:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4544.596312] LNetError: 288469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4544.608321] LNet: Removed LNI 192.168.202.113@tcp [ 4544.883574] Key type .llcrypt unregistered [ 4544.885178] Key type ._llcrypt unregistered [ 4553.453185] Key type ._llcrypt registered [ 4553.454089] Key type .llcrypt registered [ 4553.487983] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 4553.846625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4553.877329] alg: No test for adler32 (adler32-zlib) [ 4554.723247] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 4554.805082] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 4556.383124] Key type lgssc registered [ 4556.745983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4559.097982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4559.104303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4560.262539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4561.477811] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4564.051940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4564.165180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4565.346372] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4566.896381] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4567.944359] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4569.922057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4570.030620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4571.674575] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4573.264198] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4576.763799] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4577.246499] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4579.299326] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4579.300791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4579.302648] Lustre: Skipped 2 previous similar messages [ 4579.304304] Lustre: Skipped 1 previous similar message [ 4579.847667] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.13@tcp (stopping) [ 4584.416344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4584.419263] Lustre: Skipped 2 previous similar messages [ 4584.665608] Lustre: server umount lustre-MDT0000 complete [ 4584.961619] LustreError: 289842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4584.966270] LustreError: 289842:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4585.798141] LustreError: 289827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778351879 with bad export cookie 7272359647731965476 [ 4585.800912] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4585.801094] LustreError: 289827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4589.535868] LustreError: 289841:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4589.536377] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4589.546345] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4590.559831] LustreError: 289842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4590.563806] LustreError: 289842:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4595.200985] LustreError: 289840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4595.201569] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.13@tcp (stopping) [ 4595.207683] Lustre: Skipped 2 previous similar messages [ 4600.287071] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4600.321241] LustreError: 289840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4600.325843] LustreError: 289840:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4600.371767] Lustre: server umount lustre-MDT0001 complete [ 4607.705247] Lustre: server umount lustre-OST0000 complete [ 4610.416534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4611.693925] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4612.611058] Lustre: 293573:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.113@tcp [ 4613.311293] Lustre: server umount MGS complete [ 4616.001752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4616.159392] LustreError: 294143:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4616.182536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4617.440359] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4620.083504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4620.204521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4621.482172] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4624.127226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4626.003719] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4627.621232] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4628.103639] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 4629.175028] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4631.699168] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 4635.616988] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4635.618754] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4635.620543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4635.623096] Lustre: Skipped 1 previous similar message [ 4635.626013] Lustre: Skipped 3 previous similar messages [ 4638.797983] Lustre: server umount lustre-OST0000 complete [ 4640.736654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4640.736877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4640.741514] Lustre: Skipped 1 previous similar message [ 4646.041389] Lustre: server umount lustre-MDT0000 complete [ 4647.176182] LustreError: 294123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778351940 with bad export cookie 7272359647731966274 [ 4647.179642] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4647.180436] LustreError: 294123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4647.294059] Lustre: server umount lustre-MDT0001 complete [ 4650.477556] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 4651.443755] Key type lgssc unregistered [ 4651.566346] LNet: 297316:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4651.568351] LNetError: 297316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4651.577294] LNet: Removed LNI 192.168.202.113@tcp [ 4651.850797] Key type .llcrypt unregistered [ 4651.851865] Key type ._llcrypt unregistered [ 4657.225286] Key type ._llcrypt registered [ 4657.226199] Key type .llcrypt registered [ 4657.258257] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 4663.005387] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 4663.390299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4663.458806] alg: No test for adler32 (adler32-zlib) [ 4664.310044] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 4664.392425] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 4665.967153] Key type lgssc registered [ 4666.324293] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4677.111917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4677.118081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4678.215869] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4678.226460] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4678.263536] Lustre: lustre-MDT0000: new disk, initializing [ 4678.297389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4678.303740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4679.579585] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4683.619214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4683.657402] Lustre: 300412:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4683.681250] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4683.682886] Lustre: Skipped 1 previous similar message [ 4683.719302] Lustre: lustre-MDT0001: new disk, initializing [ 4683.750094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4683.761534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4683.766227] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4684.971927] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4687.933855] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4689.065959] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4691.143363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4691.241835] Lustre: lustre-OST0000: new disk, initializing [ 4691.243578] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4691.267802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4692.534211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4692.537958] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4692.568823] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4693.005457] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4695.951699] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4697.201879] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4697.273277] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4698.525119] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4698.603550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4702.690349] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4702.692183] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4702.694555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4702.695588] Lustre: Skipped 1 previous similar message [ 4705.299035] Lustre: server umount lustre-OST0000 complete [ 4707.809559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4707.810795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4707.815692] Lustre: Skipped 1 previous similar message [ 4707.818662] Lustre: Skipped 1 previous similar message [ 4712.751048] Lustre: server umount lustre-MDT0000 complete [ 4713.991443] LustreError: 300402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778352007 with bad export cookie 5266242494284514624 [ 4713.996432] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4714.121393] Lustre: server umount lustre-MDT0001 complete [ 4716.782335] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 14:40:08 (1778352008) [ 4721.625992] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 4724.410373] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 4738.214660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4738.339150] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4738.351839] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4738.407732] Lustre: lustre-MDT0000: new disk, initializing [ 4738.445346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4738.454120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4740.061734] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4744.997523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4745.033811] Lustre: 306715:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4745.054780] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4745.056183] Lustre: Skipped 1 previous similar message [ 4745.094366] Lustre: lustre-MDT0001: new disk, initializing [ 4745.124678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4745.136333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4745.139084] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4746.641106] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4750.220459] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4751.532149] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4754.170690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4754.264746] Lustre: lustre-OST0000: new disk, initializing [ 4754.266498] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4754.290676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4755.975318] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4759.412902] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4763.126066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4763.133283] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4763.183364] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4778.469966] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4778.469973] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4778.469991] LustreError: Skipped 1 previous similar message [ 4778.471906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4778.471915] Lustre: Skipped 1 previous similar message [ 4778.477814] Lustre: Skipped 1 previous similar message [ 4782.249529] Lustre: server umount lustre-OST0000 complete [ 4786.144360] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4786.149048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4786.157100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4786.160696] Lustre: Skipped 1 previous similar message [ 4790.041728] Lustre: server umount lustre-MDT0000 complete [ 4791.131023] LustreError: 306707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778352084 with bad export cookie 5266242494284515499 [ 4791.134253] LustreError: 306707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4791.134287] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4791.236713] Lustre: server umount lustre-MDT0001 complete [ 4794.252811] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 4795.397279] Key type lgssc unregistered [ 4795.561695] LNet: 310479:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4795.567385] LNetError: 310479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4795.578547] LNet: Removed LNI 192.168.202.113@tcp [ 4796.038163] Key type .llcrypt unregistered [ 4796.039512] Key type ._llcrypt unregistered [ 4827.244127] Key type ._llcrypt registered [ 4827.244964] Key type .llcrypt registered [ 4827.277878] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 4827.367207] Key type .llcrypt unregistered [ 4827.368940] Key type ._llcrypt unregistered [ 4853.565396] Key type ._llcrypt registered [ 4853.566416] Key type .llcrypt registered [ 4853.605679] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 4853.749562] Key type .llcrypt unregistered [ 4853.752103] Key type ._llcrypt unregistered [ 4864.099062] Key type ._llcrypt registered [ 4864.100181] Key type .llcrypt registered [ 4864.145212] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 4864.287272] Key type .llcrypt unregistered [ 4864.289539] Key type ._llcrypt unregistered [ 4873.961168] Key type ._llcrypt registered [ 4873.962045] Key type .llcrypt registered [ 4873.994013] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 4874.076193] Key type .llcrypt unregistered [ 4874.077139] Key type ._llcrypt unregistered [ 4898.395426] Key type ._llcrypt registered [ 4898.396284] Key type .llcrypt registered [ 4898.427902] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 4898.512273] Key type .llcrypt unregistered [ 4898.513699] Key type ._llcrypt unregistered [ 4908.224673] Key type ._llcrypt registered [ 4908.225700] Key type .llcrypt registered [ 4908.263423] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 4908.368482] Key type .llcrypt unregistered [ 4908.370163] Key type ._llcrypt unregistered [ 4918.621111] Key type ._llcrypt registered [ 4918.621996] Key type .llcrypt registered [ 4918.654762] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 4918.738269] Key type .llcrypt unregistered [ 4918.739144] Key type ._llcrypt unregistered [ 4928.731670] Key type ._llcrypt registered [ 4928.732539] Key type .llcrypt registered [ 4928.764606] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 4928.851149] Key type .llcrypt unregistered [ 4928.852069] Key type ._llcrypt unregistered [ 4938.816891] Key type ._llcrypt registered [ 4938.817840] Key type .llcrypt registered [ 4938.850444] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 4938.933210] Key type .llcrypt unregistered [ 4938.934124] Key type ._llcrypt unregistered [ 4949.503276] Key type ._llcrypt registered [ 4949.504193] Key type .llcrypt registered [ 4949.535198] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 4949.616393] Key type .llcrypt unregistered [ 4949.617335] Key type ._llcrypt unregistered [ 4970.185627] Key type ._llcrypt registered [ 4970.186558] Key type .llcrypt registered [ 4970.220689] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 4970.310452] Key type .llcrypt unregistered [ 4970.311485] Key type ._llcrypt unregistered [ 4980.741646] Key type ._llcrypt registered [ 4980.742712] Key type .llcrypt registered [ 4980.775558] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 4980.866196] Key type .llcrypt unregistered [ 4980.867193] Key type ._llcrypt unregistered [ 4994.120328] Key type ._llcrypt registered [ 4994.122799] Key type .llcrypt registered [ 4994.181380] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 4994.339479] Key type .llcrypt unregistered [ 4994.341942] Key type ._llcrypt unregistered [ 5018.947960] Key type ._llcrypt registered [ 5018.949139] Key type .llcrypt registered [ 5019.001264] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 5026.367408] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 5026.699186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5026.706733] alg: No test for adler32 (adler32-zlib) [ 5027.602205] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 5027.690671] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 5029.271095] Key type lgssc registered [ 5029.663958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5047.256496] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5047.269187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5048.384495] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 5048.398910] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 5048.440700] Lustre: tst250fs-MDT0000: new disk, initializing [ 5048.463277] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5048.470218] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5050.180591] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5055.610426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5055.677078] Lustre: 330112:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5055.708331] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 5055.712440] Lustre: Skipped 1 previous similar message [ 5055.762389] Lustre: tst250fs-MDT0001: new disk, initializing [ 5055.796874] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5055.809072] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5055.811988] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5057.527136] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5060.402556] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5063.399018] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5063.546405] Lustre: tst250fs-OST0000: new disk, initializing [ 5063.550291] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 5063.600177] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5065.918688] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5068.298314] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5068.302530] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5068.357152] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5071.464751] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5071.566168] Lustre: tst250fs-OST0001: new disk, initializing [ 5071.570082] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 5071.623507] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5074.138820] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5077.286336] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 5078.376826] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5078.380399] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5078.425145] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5091.808153] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5091.810920] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5091.815785] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 5093.859263] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 5093.861277] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5093.866699] Lustre: Skipped 2 previous similar messages [ 5095.524421] Lustre: server umount tst250fs-MDT0000 complete [ 5096.861329] LustreError: 330104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778352390 with bad export cookie 18074386797811904573 [ 5096.868739] LustreError: 330104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5096.868900] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5097.016159] Lustre: server umount tst250fs-MDT0001 complete [ 5109.200256] Lustre: server umount tst250fs-OST0000 complete [ 5121.209598] Lustre: server umount tst250fs-OST0001 complete [ 5129.408152] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 5132.893715] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 5147.609306] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 14:47:19 (1778352439) [ 5148.424343] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 5155.855453] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 5159.984456] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 5174.164304] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5057 sec ========== 14:47:45 (1778352465) [ 5174.907925] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:47:46 (1778352466) === [ 5176.478553] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:47:48 (1778352468) === [ 5190.235872] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 5191.113576] Key type lgssc unregistered [ 5191.220298] LNet: 341600:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5191.222225] LNetError: 341600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5191.228288] LNet: Removed LNI 192.168.202.113@tcp [ 5191.487076] Key type .llcrypt unregistered [ 5191.488110] Key type ._llcrypt unregistered