[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 411313853 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002353] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004007] kvm-guest: setup PV IPIs [ 0.006923] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.007010] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.008004] pid_max: default: 32768 minimum: 301 [ 0.009077] LSM: Security Framework initializing [ 0.010033] Yama: becoming mindful. [ 0.011022] SELinux: Initializing. [ 0.012037] *** VALIDATE selinux *** [ 0.019415] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.022613] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.023087] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024062] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025054] *** VALIDATE tmpfs *** [ 0.026038] *** VALIDATE proc *** [ 0.027197] *** VALIDATE cgroup *** [ 0.028006] *** VALIDATE cgroup2 *** [ 0.029208] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.031026] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.033022] Spectre V2 : User space: Vulnerable [ 0.034005] Speculative Store Bypass: Vulnerable [ 0.037580] debug: unmapping init [mem 0xffffffffb6c59000-0xffffffffb6c60fff] [ 0.039162] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040588] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041013] ... version: 2 [ 0.042007] ... bit width: 48 [ 0.043008] ... generic registers: 4 [ 0.044007] ... value mask: 0000ffffffffffff [ 0.045008] ... max period: 00007fffffffffff [ 0.046008] ... fixed-purpose events: 3 [ 0.047006] ... event mask: 000000070000000f [ 0.048171] rcu: Hierarchical SRCU implementation. [ 0.050471] smp: Bringing up secondary CPUs ... [ 0.051526] x86: Booting SMP configuration: [ 0.052016] .... node #0, CPUs: #1 #2 #3 [ 0.054841] smp: Brought up 1 node, 4 CPUs [ 0.056012] smpboot: Max logical packages: 1 [ 0.056887] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.271000] node 0 deferred pages initialised in 214ms [ 0.274019] devtmpfs: initialized [ 0.275184] x86/mm: Memory block size: 128MB [ 0.276283] gcov: version magic: 0x41383552 [ 0.279107] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.280063] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.281194] pinctrl core: initialized pinctrl subsystem [ 0.282092] [ 0.282465] ************************************************************* [ 0.284009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.285005] ** ** [ 0.287007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.288005] ** ** [ 0.289006] ** This means that this kernel is built to expose internal ** [ 0.291006] ** IOMMU data structures, which may compromise security on ** [ 0.292005] ** your system. ** [ 0.293008] ** ** [ 0.295007] ** If you see this message and you are not debugging the ** [ 0.296005] ** kernel, report this immediately to your vendor! ** [ 0.297006] ** ** [ 0.299007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.300006] ************************************************************* [ 0.301400] NET: Registered protocol family 16 [ 0.303267] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.304025] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.306026] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.308232] cpuidle: using governor menu [ 0.309326] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.311115] PCI: Using configuration type 1 for base access [ 0.312103] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.318046] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.319025] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.321061] cryptd: max_cpu_qlen set to 1000 [ 0.322187] ACPI: Added _OSI(Module Device) [ 0.323007] ACPI: Added _OSI(Processor Device) [ 0.323867] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.324007] ACPI: Added _OSI(Processor Aggregator Device) [ 0.327088] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.332406] ACPI: Interpreter enabled [ 0.333039] ACPI: PM: (supports S0 S3 S4 S5) [ 0.333903] ACPI: Using IOAPIC for interrupt routing [ 0.335051] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.337236] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.343870] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.345017] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.346009] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.348033] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.351857] acpiphp: Slot [2] registered [ 0.352117] acpiphp: Slot [5] registered [ 0.352982] acpiphp: Slot [6] registered [ 0.354069] acpiphp: Slot [7] registered [ 0.354912] acpiphp: Slot [8] registered [ 0.356060] acpiphp: Slot [9] registered [ 0.356959] acpiphp: Slot [10] registered [ 0.358061] acpiphp: Slot [3] registered [ 0.358867] acpiphp: Slot [4] registered [ 0.359044] acpiphp: Slot [11] registered [ 0.360008] acpiphp: Slot [12] registered [ 0.360878] acpiphp: Slot [13] registered [ 0.362053] acpiphp: Slot [14] registered [ 0.362937] acpiphp: Slot [15] registered [ 0.364044] acpiphp: Slot [16] registered [ 0.364847] acpiphp: Slot [17] registered [ 0.365110] acpiphp: Slot [18] registered [ 0.367095] acpiphp: Slot [19] registered [ 0.369077] acpiphp: Slot [20] registered [ 0.370093] acpiphp: Slot [21] registered [ 0.372074] acpiphp: Slot [22] registered [ 0.374066] acpiphp: Slot [23] registered [ 0.375070] acpiphp: Slot [24] registered [ 0.377084] acpiphp: Slot [25] registered [ 0.378070] acpiphp: Slot [26] registered [ 0.380093] acpiphp: Slot [27] registered [ 0.381063] acpiphp: Slot [28] registered [ 0.381904] acpiphp: Slot [29] registered [ 0.383053] acpiphp: Slot [30] registered [ 0.383893] acpiphp: Slot [31] registered [ 0.384044] PCI host bridge to bus 0000:00 [ 0.384840] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.387013] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.388008] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.390009] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.391021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.393011] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.394167] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.395753] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.397819] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.406012] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.411094] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.413011] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.415011] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.418012] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.420600] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.422560] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.424019] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.426446] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.429008] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.438009] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.442578] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.446182] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.449016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.454009] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.462009] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.471142] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.474008] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.479009] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.489014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.494665] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.499008] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.503008] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.513010] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.519055] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.522019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.526008] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.535008] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.542225] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.544954] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.548008] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.556981] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.562018] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.565885] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.569008] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.579007] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.584596] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.587214] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.588361] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.590289] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.593149] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.598107] iommu: Default domain type: Passthrough [ 0.599361] SCSI subsystem initialized [ 0.599967] ACPI: bus type USB registered [ 0.601066] usbcore: registered new interface driver usbfs [ 0.602029] usbcore: registered new interface driver hub [ 0.603032] usbcore: registered new device driver usb [ 0.604091] pps_core: LinuxPPS API ver. 1 registered [ 0.604996] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.607020] PTP clock support registered [ 0.608131] EDAC MC: Ver: 3.0.0 [ 0.610118] PCI: Using ACPI for IRQ routing [ 0.611844] NetLabel: Initializing [ 0.613008] NetLabel: domain hash size = 128 [ 0.615009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.617052] NetLabel: unlabeled traffic allowed by default [ 0.619039] vgaarb: loaded [ 0.620300] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.622012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.627352] clocksource: Switched to clocksource kvm-clock [ 0.724721] VFS: Disk quotas dquot_6.6.0 [ 0.725610] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.727108] *** VALIDATE ramfs *** [ 0.727857] *** VALIDATE hugetlbfs *** [ 0.728744] pnp: PnP ACPI init [ 0.730233] pnp: PnP ACPI: found 6 devices [ 0.743146] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.745046] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.746269] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.747579] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.748934] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.750256] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.751872] NET: Registered protocol family 2 [ 0.753320] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.756414] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.758473] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.762108] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.764116] TCP: Hash tables configured (established 65536 bind 65536) [ 0.765789] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.767616] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.770445] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.773382] NET: Registered protocol family 1 [ 0.775767] RPC: Registered named UNIX socket transport module. [ 0.778029] RPC: Registered udp transport module. [ 0.779842] RPC: Registered tcp transport module. [ 0.781599] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.783917] NET: Registered protocol family 44 [ 0.785600] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.787893] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.790074] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.792576] PCI: CLS 0 bytes, default 64 [ 0.794211] Unpacking initramfs... [ 2.053674] debug: unmapping init [mem 0xffff9750bcc54000-0xffff9750bffbffff] [ 2.058230] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.059590] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.061281] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.483904] Initialise system trusted keyrings [ 2.485026] Key type blacklist registered [ 2.486250] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.496932] zbud: loaded [ 2.500065] *** VALIDATE nfs *** [ 2.500905] *** VALIDATE nfs4 *** [ 2.501919] pstore: using deflate compression [ 2.510576] Platform Keyring initialized [ 2.579758] NET: Registered protocol family 38 [ 2.580867] Key type asymmetric registered [ 2.581822] Asymmetric key parser 'x509' registered [ 2.582922] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.584850] io scheduler mq-deadline registered [ 2.586088] io scheduler kyber registered [ 2.587350] io scheduler bfq registered [ 2.588466] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.590258] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.592138] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.593863] ACPI: Power Button [PWRF] [ 2.598575] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.602945] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.610259] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.614486] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.623346] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.648160] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.674053] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.678161] Non-volatile memory driver v1.3 [ 2.679832] Linux agpgart interface v0.103 [ 2.709743] virtio_blk virtio1: [vda] 134736 512-byte logical blocks (69.0 MB/65.8 MiB) [ 2.713936] vda: detected capacity change from 0 to 68984832 [ 2.723819] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.725650] vdb: detected capacity change from 0 to 1073741824 [ 2.737438] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.739284] vdc: detected capacity change from 0 to 2621440000 [ 2.749962] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.751843] vdd: detected capacity change from 0 to 2621440000 [ 2.762923] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.766214] vde: detected capacity change from 0 to 4294967296 [ 2.780452] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.783933] vdf: detected capacity change from 0 to 4294967296 [ 2.789967] libphy: Fixed MDIO Bus: probed [ 2.794906] usbcore: registered new interface driver usbserial_generic [ 2.797603] usbserial: USB Serial support registered for generic [ 2.800174] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.804749] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.806805] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.809340] mousedev: PS/2 mouse device common for all mice [ 2.812273] rtc_cmos 00:05: RTC can wake from S4 [ 2.815044] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.818442] rtc_cmos 00:05: registered as rtc0 [ 2.819902] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.821207] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.826296] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.828060] intel_pstate: CPU model not supported [ 2.835316] hid: raw HID events driver (C) Jiri Kosina [ 2.837829] usbcore: registered new interface driver usbhid [ 2.840432] usbhid: USB HID core driver [ 2.842193] drop_monitor: Initializing network drop monitor service [ 2.844990] Initializing XFRM netlink socket [ 2.847316] NET: Registered protocol family 10 [ 2.850329] Segment Routing with IPv6 [ 2.851766] NET: Registered protocol family 17 [ 2.853744] mpls_gso: MPLS GSO support [ 2.858656] RAS: Correctable Errors collector initialized. [ 2.860797] AVX version of gcm_enc/dec engaged. [ 2.862708] AES CTR mode by8 optimization enabled [ 2.920058] sched_clock: Marking stable (2919916716, 0)->(3696917273, -777000557) [ 2.922423] registered taskstats version 1 [ 2.923620] Loading compiled-in X.509 certificates [ 2.926212] zswap: loaded using pool lzo/zbud [ 2.944569] Key type big_key registered [ 2.955490] Key type encrypted registered [ 2.957598] ima: No TPM chip found, activating TPM-bypass! [ 2.958752] ima: Allocated hash algorithm: sha1 [ 2.959706] ima: No architecture policies found [ 2.960745] evm: Initialising EVM extended attributes: [ 2.961812] evm: security.selinux [ 2.962534] evm: security.ima [ 2.963146] evm: security.capability [ 2.963873] evm: HMAC attrs: 0x1 [ 2.967451] rtc_cmos 00:05: setting system clock to 2026-02-27 04:41:17 UTC (1772167277) [ 2.971344] debug: unmapping init [mem 0xffffffffb7c03000-0xffffffffb7dfffff] [ 2.973086] debug: unmapping init [mem 0xffffffffb6982000-0xffffffffb6c58fff] [ 2.980096] Write protecting the kernel read-only data: 28672k [ 2.982158] debug: unmapping init [mem 0xffffffffb5003000-0xffffffffb51fffff] [ 2.983760] debug: unmapping init [mem 0xffffffffb5914000-0xffffffffb59fffff] [ 3.011206] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.017945] systemd[1]: Detected virtualization kvm. [ 3.019471] systemd[1]: Detected architecture x86-64. [ 3.021203] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.043635] systemd[1]: No hostname configured. [ 3.045256] systemd[1]: Set hostname to . [ 3.046921] random: systemd: uninitialized urandom read (16 bytes read) [ 3.048992] systemd[1]: Initializing machine ID from random generator. [ 3.083612] random: ln: uninitialized urandom read (6 bytes read) [ 3.152160] random: systemd: uninitialized urandom read (16 bytes read) [ 3.153830] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.157410] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.163340] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ 3.291117] random: fast init done [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.669860] device-mapper: uevent: version 1.0.3 [ 3.671322] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.304868] virtio_net virtio0 ens2: renamed from eth0 [ 4.339283] scsi host0: ata_piix [ 4.384218] scsi host1: ata_piix [ 4.386343] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.389349] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.664869] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.470223] random: crng init done [ 9.471407] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.648593] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.516223] printk: systemd: 26 output lines suppressed due to ratelimiting [ 10.670957] SELinux: Disabled at runtime. [ 10.714578] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 10.720748] systemd[1]: Detected virtualization kvm. [ 10.722113] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.035655] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.039549] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.044690] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.048302] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.051545] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.056318] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.062202] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ 11.213469] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Col[ 11.475452] squashfs: version 4.0 (2009/01/31) Phillip Lougher dplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 11.774869] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 11.792384] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 11.935251] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 11.954209] EDAC sbridge: Ver: 1.1.2 [ 13.340767] Key type dns_resolver registered [ 13.657507] NFS: Registering the id_resolver key type [ 13.658989] Key type id_resolver registered [ 13.660254] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Crash recovery kernel arming. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg428-server login: [ 37.230486] libcfs: loading out-of-tree module taints kernel. [ 37.248061] Key type ._llcrypt registered [ 37.249066] Key type .llcrypt registered [ 37.291715] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 44.902853] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 45.617116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 45.621925] alg: No test for adler32 (adler32-zlib) [ 46.563133] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 46.817792] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 48.415122] Key type lgssc registered [ 48.900772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 55.194374] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 70.385528] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 75.505676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 75.514732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 76.634674] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 76.645885] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 76.682678] Lustre: lustre-MDT0000: new disk, initializing [ 76.708858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 76.715393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 77.990595] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 84.257914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 84.306447] Lustre: 6473:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 84.325756] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 84.332258] Lustre: Skipped 1 previous similar message [ 84.385158] Lustre: lustre-MDT0001: new disk, initializing [ 84.410438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 84.422827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 84.430644] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 85.804341] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 88.367036] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 92.106274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 92.228079] Lustre: lustre-OST0000: new disk, initializing [ 92.230265] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 92.253947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 94.554441] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 101.358158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 101.362719] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 101.389733] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 101.614327] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 101.686493] Lustre: lustre-OST0001: new disk, initializing [ 101.690002] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 101.715175] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 104.039144] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 110.101046] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 111.094643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 111.099340] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 111.138533] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 113.150733] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 120.654249] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing check_logdir /tmp/testlogs/ [ 122.691141] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing yml_node [ 124.664157] Lustre: DEBUG MARKER: Client: 2.17.50.152 [ 125.697939] Lustre: DEBUG MARKER: MDS: 2.17.50.152 [ 126.598752] Lustre: DEBUG MARKER: OSS: 2.17.50.152 [ 127.204377] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Feb 26 23:43:20 EST 2026 [ 135.352383] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 136.047585] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 151.007687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 151.010328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 151.015157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 152.031777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 152.032072] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 152.034371] Lustre: Skipped 2 previous similar messages [ 152.038323] Lustre: Skipped 2 previous similar messages [ 157.152321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 157.156474] Lustre: Skipped 4 previous similar messages [ 162.271149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 162.272260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 162.281572] Lustre: Skipped 3 previous similar messages [ 162.369649] Lustre: server umount lustre-MDT0000 complete [ 164.034125] LustreError: 6463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772167438 with bad export cookie 13762256526022271473 [ 164.036318] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 164.040858] LustreError: 6463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 167.391883] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 167.392408] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 167.397668] Lustre: Skipped 1 previous similar message [ 167.401867] Lustre: Skipped 1 previous similar message [ 170.249576] Lustre: server umount lustre-MDT0001 complete [ 172.110860] Lustre: server umount lustre-OST0000 complete [ 173.691935] Lustre: server umount lustre-OST0001 complete [ 176.185719] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 180.410932] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 195.892419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 196.028673] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 196.046342] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 196.085170] Lustre: lustre-MDT0000: new disk, initializing [ 196.109873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 196.116800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 197.696941] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 202.729459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 202.768788] Lustre: 14940:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 202.880432] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 204.448010] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 207.960714] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 209.644673] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 212.530734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 212.710278] Lustre: lustre-OST0000: new disk, initializing [ 212.712617] Lustre: Skipped 1 previous similar message [ 212.715626] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 212.719692] Lustre: Skipped 2 previous similar messages [ 212.772350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 212.777634] Lustre: Skipped 1 previous similar message [ 214.479797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 214.485533] Lustre: Skipped 1 previous similar message [ 214.489635] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 214.553739] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 215.154414] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 218.875801] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 220.748849] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 220.848824] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 222.561708] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 222.634324] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 224.740476] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 224.741283] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 224.745551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 224.747941] Lustre: Skipped 1 previous similar message [ 224.750420] Lustre: Skipped 2 previous similar messages [ 229.400939] Lustre: server umount lustre-OST0000 complete [ 233.952442] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 233.956747] LustreError: Skipped 1 previous similar message [ 233.959300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 237.174298] Lustre: server umount lustre-MDT0000 complete [ 238.797605] LustreError: 15718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772167513 with bad export cookie 13762256526022273006 [ 238.803231] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 238.957526] Lustre: server umount lustre-MDT0001 complete [ 241.548250] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 23:45:14 (1772167514) [ 250.018435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 250.228211] LustreError: 19406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 250.233080] LustreError: 19406:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 250.250390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 251.739993] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 255.458044] LustreError: 19406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 255.608166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 257.298516] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 259.488456] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 261.100971] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 261.975774] Lustre: Failing over lustre-MDT0000 [ 262.068103] Lustre: server umount lustre-MDT0000 complete [ 265.185383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 265.186587] LustreError: 20861:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 265.193029] Lustre: Skipped 2 previous similar messages [ 265.204805] LustreError: 20861:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 271.328867] LustreError: 19406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 271.337509] LustreError: 19406:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 275.814165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 275.871543] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 277.535705] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 281.057035] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 281.059531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 281.065581] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 281.321882] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 286.176576] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 286.181650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 286.189845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 286.193536] Lustre: Skipped 4 previous similar messages [ 289.841657] Lustre: server umount lustre-MDT0000 complete [ 291.706302] LustreError: 19386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772167566 with bad export cookie 13762256526022274420 [ 291.711155] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 291.711212] LustreError: 19386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 291.849935] Lustre: server umount lustre-MDT0001 complete [ 295.640072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 295.897927] LustreError: 22399:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 295.929117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 295.931811] Lustre: Skipped 2 previous similar messages [ 297.544967] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 300.626549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 302.448529] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 304.893397] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 306.346549] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 307.244756] Lustre: Failing over lustre-MDT0000 [ 307.326857] Lustre: server umount lustre-MDT0000 complete [ 311.265549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 311.266732] LustreError: 23109:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 311.277724] LustreError: 23109:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 320.726835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 320.773214] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 322.215993] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 325.314337] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 326.113979] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 326.118913] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 326.123596] Lustre: Skipped 1 previous similar message [ 326.131718] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 331.232562] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 331.237234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 331.244484] Lustre: Skipped 3 previous similar messages [ 331.247768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 331.253829] Lustre: Skipped 2 previous similar messages [ 333.747996] Lustre: server umount lustre-MDT0000 complete [ 335.491928] LustreError: 22379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772167610 with bad export cookie 13762256526022275358 [ 335.498757] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 335.499628] LustreError: 22379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 335.626899] Lustre: server umount lustre-MDT0001 complete [ 339.813509] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 23:46:52 (1772167612) [ 346.147408] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 350.829941] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 367.483829] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 372.259556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 372.387440] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 372.408558] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 372.465746] Lustre: lustre-MDT0000: new disk, initializing [ 372.498814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 372.501933] Lustre: Skipped 2 previous similar messages [ 372.510368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 374.320418] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 379.746857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 379.794951] Lustre: 28374:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 379.905541] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 381.466034] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 385.391837] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 386.888592] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 390.016584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 391.633604] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 391.674573] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 391.683436] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 392.512386] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 395.980628] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 398.683260] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 405.661433] Lustre: server umount lustre-OST0000 complete [ 410.592241] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 410.595286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 410.600310] Lustre: Skipped 1 previous similar message [ 410.602663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 410.605176] Lustre: Skipped 2 previous similar messages [ 417.295080] Lustre: server umount lustre-MDT0000 complete [ 418.882063] LustreError: 29151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772167693 with bad export cookie 13762256526022275764 [ 418.884232] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 418.885068] LustreError: 29151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 429.104475] Lustre: server umount lustre-MDT0001 complete [ 433.274006] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 434.322861] Key type lgssc unregistered [ 434.436310] LNet: 31938:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 434.438670] LNetError: 31938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 434.450287] LNet: Removed LNI 192.168.204.128@tcp [ 434.735087] Key type .llcrypt unregistered [ 434.736186] Key type ._llcrypt unregistered [ 445.513416] Key type ._llcrypt registered [ 445.515230] Key type .llcrypt registered [ 445.577289] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 446.033040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 446.039176] alg: No test for adler32 (adler32-zlib) [ 446.882497] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 446.956613] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 448.527148] Key type lgssc registered [ 449.036451] Lustre: Echo OBD driver; http://www.lustre.org/ [ 452.355244] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 452.360687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 453.502945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 455.046665] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 458.130109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 458.249919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 459.588803] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 461.646162] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 463.002590] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 466.066837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 466.231970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 468.270655] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 468.504822] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 470.572736] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 487.592495] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 23:49:20 (1772167760) [ 488.325237] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 488.931212] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 491.771248] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 23:49:25 (1772167765) [ 495.081968] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 23:49:28 (1772167768) [ 495.463650] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 498.911977] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 498.916545] Lustre: Skipped 2 previous similar messages [ 499.588723] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 23:49:32 (1772167772) [ 502.915712] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 502.918055] Lustre: Skipped 4 previous similar messages [ 509.068825] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 23:49:42 (1772167782) [ 510.047218] Lustre: *** cfs_fail_loc=131b, val=2*** [ 511.716763] Lustre: *** cfs_fail_loc=131b, val=2*** [ 515.247555] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 23:49:48 (1772167788) [ 515.879582] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 515.883564] Lustre: Skipped 9 previous similar messages [ 521.855366] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 23:49:54 (1772167794) [ 529.891449] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 529.893717] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 529.896387] LustreError: Skipped 1 previous similar message [ 529.897733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 529.903580] Lustre: Skipped 1 previous similar message [ 529.911884] Lustre: Skipped 1 previous similar message [ 535.012523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 535.016428] Lustre: Skipped 1 previous similar message [ 540.132483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 540.136507] Lustre: Skipped 1 previous similar message [ 541.151186] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 541.228253] Lustre: server umount lustre-OST0000 complete [ 545.248977] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 545.249225] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 545.251892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 545.255235] Lustre: Skipped 1 previous similar message [ 549.078531] Lustre: server umount lustre-MDT0000 complete [ 550.368825] LustreError: 34033:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.378405] LustreError: 34033:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 550.538412] LustreError: 35917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772167825 with bad export cookie 12575257270289851067 [ 550.541437] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 550.544992] LustreError: 35917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 550.688873] Lustre: server umount lustre-MDT0001 complete [ 554.495362] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 555.811209] Key type lgssc unregistered [ 555.963552] LNet: 40806:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 555.969257] LNetError: 40806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 555.978601] LNet: Removed LNI 192.168.204.128@tcp [ 556.415711] Key type .llcrypt unregistered [ 556.418056] Key type ._llcrypt unregistered [ 563.493574] Key type ._llcrypt registered [ 563.494951] Key type .llcrypt registered [ 563.543019] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 571.253262] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 571.562877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 571.603514] alg: No test for adler32 (adler32-zlib) [ 572.449086] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 572.520255] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 574.095168] Key type lgssc registered [ 574.569570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 590.320232] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 590.331432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 591.453709] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 591.467626] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 591.506276] Lustre: lustre-MDT0000: new disk, initializing [ 591.534189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 591.543792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 593.187249] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 598.677859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 598.738145] Lustre: 43899:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 598.778267] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 598.782583] Lustre: Skipped 1 previous similar message [ 598.830906] Lustre: lustre-MDT0001: new disk, initializing [ 598.875286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 598.894849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 598.900732] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 600.622554] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 604.314872] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 605.692329] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 608.332691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 608.480140] Lustre: lustre-OST0000: new disk, initializing [ 608.482603] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 608.520448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 610.926980] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 614.584087] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 615.443042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 615.449870] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 615.500260] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 616.230230] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 616.296150] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 617.713215] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 617.773870] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 620.514228] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 620.517642] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 620.520268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 620.529125] Lustre: Skipped 1 previous similar message [ 624.498751] Lustre: server umount lustre-OST0000 complete [ 629.728827] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 629.733843] LustreError: Skipped 1 previous similar message [ 629.737101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 629.746131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 629.750273] Lustre: Skipped 1 previous similar message [ 630.753461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 630.754530] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 632.289198] Lustre: server umount lustre-MDT0000 complete [ 633.796070] LustreError: 44675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772167908 with bad export cookie 282871872191637936 [ 633.802706] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 633.805094] LustreError: 44675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 633.959161] Lustre: server umount lustre-MDT0001 complete [ 637.904724] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 23:51:50 (1772167910) [ 641.115718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 641.343191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 642.862288] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 646.386535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 646.521573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 648.076204] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 650.404945] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 651.834439] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 654.794494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 654.989555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 657.504899] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 659.937583] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 672.289891] LustreError: 50160:0:(llog.c:441:llog_init_handle()) llog type is not specified! [ 672.711227] Lustre: Modifying parameter general.timeout=1 in log params [ 674.055827] Lustre: Modifying parameter general.timeout=4 in log params [ 674.059780] Lustre: Skipped 2 previous similar messages [ 676.238124] Lustre: Modifying parameter general.timeout=9 in log params [ 676.242268] Lustre: Skipped 4 previous similar messages [ 680.473652] Lustre: Modifying parameter general.timeout=19 in log params [ 680.477819] Lustre: Skipped 9 previous similar messages [ 688.675222] Lustre: Modifying parameter general.timeout=39 in log params [ 688.679063] Lustre: Skipped 19 previous similar messages [ 704.894578] Lustre: Modifying parameter general.timeout=82 in log params [ 704.896458] Lustre: Skipped 42 previous similar messages [ 724.682961] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 23:53:18 (1772167998) [ 725.813416] Lustre: 57063:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 727.259352] Lustre: 57260:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 727.263453] Lustre: 57260:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 729.281778] Lustre: 57555:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 729.291060] Lustre: 57555:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 2 previous similar messages [ 737.399218] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 in log params [ 737.405411] Lustre: Skipped 67 previous similar messages [ 740.419448] Lustre: 58903:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 740.426919] LustreError: 58903:0:(mgs_handler.c:1164:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 742.141765] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 23:53:35 (1772168015) [ 747.489835] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 747.491069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 747.494952] Lustre: Skipped 2 previous similar messages [ 747.498926] Lustre: Skipped 3 previous similar messages [ 752.609242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 752.615121] Lustre: Skipped 2 previous similar messages [ 753.137636] Lustre: server umount lustre-MDT0000 complete [ 754.589815] LustreError: 47929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772168029 with bad export cookie 282871872191638804 [ 754.595801] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 754.598559] LustreError: 47929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 754.606345] LustreError: 47940:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 754.736654] Lustre: server umount lustre-MDT0001 complete [ 781.402680] Lustre: server umount lustre-OST0000 complete [ 793.020258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 793.176933] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 793.330982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 794.777786] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 798.723189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 798.778383] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 798.794564] Lustre: 61431:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 800.562662] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 802.503722] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 803.870447] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 806.905543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 807.023346] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 809.426898] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 811.653624] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 812.396275] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 812.403846] Lustre: Skipped 3 previous similar messages [ 819.171200] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 819.172388] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 819.177251] LustreError: Skipped 1 previous similar message [ 819.186571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 819.193124] Lustre: Skipped 1 previous similar message [ 821.096076] Lustre: server umount lustre-OST0000 complete [ 837.087136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 837.165144] Lustre: server umount lustre-MDT0000 complete [ 838.625742] LustreError: 63615:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 838.632695] LustreError: 63615:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 838.989731] LustreError: 61423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772168113 with bad export cookie 282871872191641226 [ 838.994270] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 838.997567] LustreError: 61423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 839.154693] Lustre: server umount lustre-MDT0001 complete [ 843.400803] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 844.391913] Key type lgssc unregistered [ 844.519367] LNet: 64845:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.522671] LNetError: 64845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 844.530372] LNet: Removed LNI 192.168.204.128@tcp [ 844.851435] Key type .llcrypt unregistered [ 844.853252] Key type ._llcrypt unregistered [ 855.545099] Key type ._llcrypt registered [ 855.546374] Key type .llcrypt registered [ 855.597371] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 856.009659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 856.050064] alg: No test for adler32 (adler32-zlib) [ 856.936208] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 857.026080] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 858.615129] Key type lgssc registered [ 859.153679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 862.239243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 862.251211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 863.478059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 864.848377] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 868.313303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 868.566341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 869.949029] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 871.824882] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 872.936885] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 875.950277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 876.204715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 878.407939] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 880.343388] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 894.000669] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 899.043508] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 899.047356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 899.052149] Lustre: Skipped 2 previous similar messages [ 899.055954] Lustre: Skipped 1 previous similar message [ 903.353752] Lustre: server umount lustre-MDT0000 complete [ 904.160793] LustreError: 66217:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 904.168735] LustreError: 66217:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 904.716704] LustreError: 66196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772168179 with bad export cookie 4254944197818603420 [ 904.721832] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 904.723655] LustreError: 66196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 904.880657] Lustre: server umount lustre-MDT0001 complete [ 916.919508] Lustre: server umount lustre-OST0000 complete [ 926.943961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 927.088261] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 927.099865] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 927.197761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 928.695086] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 932.608992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 932.659371] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 932.676597] Lustre: 70772:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 932.873408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 934.462784] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 936.461358] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 937.787307] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 940.965302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 941.069899] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 941.235130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 943.737236] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 945.696930] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 946.415315] Lustre: 73006:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 947.292759] Lustre: Modifying parameter general.jobid_var=disable in log params [ 950.753937] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 950.758230] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 950.759392] LustreError: Skipped 1 previous similar message [ 950.769499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 950.774863] Lustre: Skipped 2 previous similar messages [ 954.731089] Lustre: server umount lustre-OST0000 complete [ 958.432498] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 958.437659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 958.444921] Lustre: Skipped 1 previous similar message [ 958.447722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 960.993272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 960.993978] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 962.295761] Lustre: server umount lustre-MDT0000 complete [ 963.655322] LustreError: 70810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772168238 with bad export cookie 4254944197818604239 [ 963.658259] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 963.659014] LustreError: 70810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 963.769906] Lustre: server umount lustre-MDT0001 complete [ 967.960297] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 969.236634] Key type lgssc unregistered [ 969.373652] LNet: 74198:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 969.375772] LNetError: 74198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 969.385460] LNet: Removed LNI 192.168.204.128@tcp [ 969.769532] Key type .llcrypt unregistered [ 969.771563] Key type ._llcrypt unregistered [ 980.979519] Key type ._llcrypt registered [ 980.981200] Key type .llcrypt registered [ 981.023698] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 981.348308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 981.396614] alg: No test for adler32 (adler32-zlib) [ 982.254018] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 982.334352] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 983.919089] Key type lgssc registered [ 984.224727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 987.512809] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 987.524318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 988.727940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 990.179599] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 993.479988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 993.755376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 995.225593] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 997.047804] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 998.504405] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1001.650589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1001.899262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1003.844155] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1006.108759] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1018.771592] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1019.556897] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1019.558533] Lustre: Skipped 1 previous similar message [ 1020.892047] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1020.893746] Lustre: Skipped 3 previous similar messages [ 1022.932529] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1022.934356] Lustre: Skipped 5 previous similar messages [ 1027.140902] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1027.142671] Lustre: Skipped 11 previous similar messages [ 1035.311312] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1035.313068] Lustre: Skipped 22 previous similar messages [ 1051.369953] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1051.372331] Lustre: Skipped 43 previous similar messages [ 1058.019226] Lustre: 83331:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1059.131154] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 23:58:52 (1772168332) [ 1059.899849] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1060.575772] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 23:58:53 (1772168333) [ 1065.442882] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1065.447873] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1065.453293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1065.455441] Lustre: Skipped 1 previous similar message [ 1070.565321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1070.569337] Lustre: Skipped 2 previous similar messages [ 1075.685836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1075.690807] Lustre: Skipped 1 previous similar message [ 1076.191080] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1076.268664] Lustre: server umount lustre-OST0000 complete [ 1080.800781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1080.801371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1080.804727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1080.805920] LustreError: Skipped 1 previous similar message [ 1080.807416] Lustre: Skipped 1 previous similar message [ 1084.118726] Lustre: server umount lustre-MDT0000 complete [ 1085.629187] LustreError: 75553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772168360 with bad export cookie 17546156958775889046 [ 1085.635935] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1085.636734] LustreError: 75553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1085.754225] Lustre: server umount lustre-MDT0001 complete [ 1089.925303] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 1091.324675] Key type lgssc unregistered [ 1091.434339] LNet: 84561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1091.436522] LNetError: 84561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1091.447337] LNet: Removed LNI 192.168.204.128@tcp [ 1091.697888] Key type .llcrypt unregistered [ 1091.698864] Key type ._llcrypt unregistered [ 1093.165474] Key type ._llcrypt registered [ 1093.166875] Key type .llcrypt registered [ 1093.224907] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1094.990962] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules [ 1095.402907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1095.416318] alg: No test for adler32 (adler32-zlib) [ 1096.270193] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 1096.367855] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 1097.951109] Key type lgssc registered [ 1098.312468] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1098.653408] LustreError: 85680:0:(lu_object.c:1434:lu_context_key_register()) cfs_race id 60d sleeping [ 1108.043760] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1112.016061] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1120.151687] LustreError: 85680:0:(lu_object.c:1434:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1120.166361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1121.373085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1122.937643] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1126.555682] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 23:59:59 (1772168399) [ 1128.508967] LustreError: 86717:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1128.512336] LustreError: 86717:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1128.515015] LustreError: 86717:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 1128.649519] Lustre: server umount lustre-MDT0000 complete [ 1134.172547] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 1135.466615] Key type lgssc unregistered [ 1135.595387] LNet: 88178:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1135.597336] LNetError: 88178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1135.605463] LNet: Removed LNI 192.168.204.128@tcp [ 1135.876718] Key type .llcrypt unregistered [ 1135.877670] Key type ._llcrypt unregistered [ 1143.616702] Key type ._llcrypt registered [ 1143.617891] Key type .llcrypt registered [ 1143.660678] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1144.173994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1144.191492] alg: No test for adler32 (adler32-zlib) [ 1145.064684] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 1145.174741] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 1146.775152] Key type lgssc registered [ 1147.204442] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1150.564547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1150.571090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1151.848180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1153.409320] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1156.501439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1156.745677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1158.373249] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1160.564154] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1162.071432] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1164.760072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1165.045893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1167.447948] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1169.676289] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1204.137205] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 00:01:17 (1772168477) [ 1204.923521] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1205.865230] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 00:01:18 (1772168478) [ 1207.776691] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1207.781761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1207.789694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1208.295375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1208.295835] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1208.299483] Lustre: Skipped 1 previous similar message [ 1208.311078] Lustre: Skipped 1 previous similar message [ 1213.382144] Lustre: server umount lustre-MDT0000 complete [ 1213.412434] LustreError: 89358:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1213.417186] LustreError: 89358:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1215.166531] LustreError: 92208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772168489 with bad export cookie 2990742574373385924 [ 1215.172154] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1215.174133] LustreError: 92208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1215.360199] Lustre: server umount lustre-MDT0001 complete [ 1227.618535] Lustre: server umount lustre-OST0000 complete [ 1233.207614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1233.423563] LustreError: 93326:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1233.519698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1235.392968] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1239.009134] LustreError: 93326:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1239.715228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1240.006520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1241.607961] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1244.204541] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1245.420101] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1251.245541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1251.352999] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1251.358487] LustreError: 93315:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1251.377708] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1251.381482] LustreError: 95181:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -98 [ 1251.384686] LustreError: 95181:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 1251.387354] LustreError: 95181:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1251.424111] Lustre: server umount lustre-OST0000 complete [ 1251.427098] LustreError: 95181:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1254.805460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1254.891476] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1254.895030] LustreError: 93315:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1254.898433] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1254.903289] LustreError: 95643:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -98 [ 1254.907175] LustreError: 95643:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 1254.910647] LustreError: 95643:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1254.938772] Lustre: server umount lustre-OST0000 complete [ 1254.941112] LustreError: 95643:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1257.851410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1257.914253] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1257.915718] Lustre: Found index 0 for lustre-OST0000, updating log [ 1257.917520] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1257.942337] Lustre: lustre-OST0000: new disk, initializing [ 1257.944532] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1258.023306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1260.014279] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1261.702323] Lustre: Failing over lustre-OST0000 [ 1261.803925] Lustre: server umount lustre-OST0000 complete [ 1264.098515] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1264.099457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1264.105989] Lustre: Skipped 1 previous similar message [ 1264.109664] Lustre: Skipped 1 previous similar message [ 1270.241388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1270.245453] Lustre: Skipped 3 previous similar messages [ 1273.336879] Lustre: server umount lustre-MDT0000 complete [ 1274.806101] LustreError: 93306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772168549 with bad export cookie 2990742574373388647 [ 1274.812162] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1274.813760] LustreError: 93306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1274.962722] Lustre: server umount lustre-MDT0001 complete [ 1279.066846] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 00:02:32 (1772168552) [ 1286.076593] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1290.341832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1290.526819] LustreError: 98487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1290.561461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1292.121929] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1295.597932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1297.562260] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1298.738428] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1301.706612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1302.040883] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1304.176553] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1306.145552] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1306.149679] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:10 [ 1306.154562] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 1306.161159] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1306.161261] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 1306.163708] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 1307.217661] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1307.248734] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1307.281119] Lustre: lustre-OST0001: new disk, initializing [ 1307.282981] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1309.906236] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1313.594288] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1315.700891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1315.707889] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1315.765858] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1319.548315] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1326.561279] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1326.566300] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1326.574890] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1331.175418] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1331.183311] Lustre: Skipped 1 previous similar message [ 1332.653941] Lustre: server umount lustre-MDT0001 complete [ 1336.293683] LustreError: 98487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1336.303745] LustreError: 98487:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1337.232543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1337.292175] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1337.296218] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1337.325651] Lustre: 98475:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1337.633931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1337.636309] Lustre: Skipped 3 previous similar messages [ 1339.303262] LustreError: 102429:0:(obd_mount.c:212:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1339.307208] LustreError: 102429:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1339.310909] LustreError: 98478:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1339.343134] LustreError: 102443:0:(obd_config.c:2136:class_config_llog_handler()) MGC192.168.204.128@tcp: cfg command failed: rc = -17 [ 1339.346918] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1339.346918] [ 1339.543175] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1342.960624] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1343.948755] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 00:03:37 (1772168617) [ 1351.910658] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1359.513134] Lustre: 104312:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1370.279457] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1371.827543] Lustre: 105443:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1429.986155] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1429.986800] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1429.988331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1429.988342] Lustre: Skipped 5 previous similar messages [ 1430.011979] Lustre: Skipped 3 previous similar messages [ 1434.562585] Lustre: server umount lustre-MDT0000 complete [ 1435.105107] LustreError: 98483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1435.115059] LustreError: 98483:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1436.246555] LustreError: 105800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772168710 with bad export cookie 2990742574373389529 [ 1436.249507] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1436.253639] LustreError: 105800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1436.414437] Lustre: server umount lustre-MDT0001 complete [ 1448.455419] Lustre: server umount lustre-OST0000 complete [ 1460.480763] Lustre: server umount lustre-OST0001 complete [ 1463.104276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1467.140349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1471.075412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1475.263141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1483.820192] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1488.353296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1488.365610] Lustre: lustre-MDT0000: reset Object Index mappings [ 1488.584421] LustreError: 109375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1488.624757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1490.160408] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1493.679140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1493.688206] Lustre: lustre-MDT0001: reset Object Index mappings [ 1495.766942] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1497.281465] Lustre: 110487:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1500.081528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1500.676780] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:382 to 0x280000401:418) [ 1503.168181] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1503.800185] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:199 to 0x280000400:225) [ 1507.422471] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1509.189773] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 1509.192296] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 1510.499230] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1514.701739] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1516.515548] Lustre: 112337:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1543.931702] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 00:06:57 (1772168817) [ 1574.370679] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1574.371961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1574.376667] Lustre: Skipped 2 previous similar messages [ 1574.380152] Lustre: Skipped 6 previous similar messages [ 1578.733251] Lustre: server umount lustre-MDT0000 complete [ 1579.491412] LustreError: 109369:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1579.496458] LustreError: 109369:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 1579.951340] LustreError: 109356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772168854 with bad export cookie 2990742574373436450 [ 1579.957903] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1594.335127] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1594.435745] Lustre: server umount lustre-MDT0001 complete [ 1610.207158] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1610.251189] Lustre: server umount lustre-OST0000 complete [ 1617.693969] Lustre: server umount lustre-OST0001 complete [ 1619.802842] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 1622.738141] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1634.273342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1634.358596] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1634.371091] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1634.404783] Lustre: lustre-MDT0000: new disk, initializing [ 1634.427836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1634.430423] Lustre: Skipped 3 previous similar messages [ 1634.435209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1635.661473] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1638.494677] Lustre: server umount lustre-MDT0000 complete [ 1641.502757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1641.603209] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1641.607667] Lustre: 116547:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1641.613394] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1642.998938] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1644.660437] Lustre: server umount lustre-MDT0000 complete [ 1647.454349] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 00:08:40 (1772168920) [ 1647.960441] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1648.584394] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 00:08:41 (1772168921) [ 1654.233722] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1657.619966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1659.157340] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1661.959661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1662.001762] Lustre: 118563:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1662.025222] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1662.027535] Lustre: Skipped 1 previous similar message [ 1662.061296] Lustre: lustre-MDT0001: new disk, initializing [ 1662.104966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1662.108487] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1663.341504] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1665.676828] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1667.927726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1668.049858] Lustre: lustre-OST0000: new disk, initializing [ 1668.051655] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1668.566345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1668.570565] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1668.610857] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1669.971428] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1674.198399] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1675.388898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1675.392161] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1675.425258] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1676.166486] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1680.574695] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1681.819552] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1835.847196] Lustre: 127312:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 1835.853455] Lustre: 127312:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 1835.858143] Lustre: 127312:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 1835.862911] Lustre: 127312:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 1835.867950] Lustre: 127312:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 1835.872206] Lustre: 127312:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1918.737806] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 00:13:11 (1772169191) [ 1931.748863] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1931.752501] Lustre: Skipped 6 previous similar messages [ 1931.753773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1931.755352] Lustre: Skipped 6 previous similar messages [ 1936.448606] Lustre: server umount lustre-MDT0000 complete [ 1936.865127] LustreError: 118573:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1936.870504] LustreError: 118573:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 1937.677322] LustreError: 118554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772169212 with bad export cookie 2990742574373494249 [ 1937.680312] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1937.680454] LustreError: 118554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1937.825077] Lustre: server umount lustre-MDT0001 complete [ 1955.268532] Lustre: server umount lustre-OST0000 complete [ 1971.804270] Lustre: server umount lustre-OST0001 complete [ 1973.755443] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 1977.062619] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1989.303562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1989.410355] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1989.421641] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1989.423875] Lustre: Skipped 1 previous similar message [ 1989.464200] Lustre: lustre-MDT0000: new disk, initializing [ 1989.465560] Lustre: Skipped 1 previous similar message [ 1989.491849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1989.493765] Lustre: Skipped 5 previous similar messages [ 1989.499872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1990.851453] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1994.955771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1994.996593] Lustre: 130313:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1995.116580] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1996.430323] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1999.508181] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2000.651447] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2002.942199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2004.344392] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2004.382058] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2004.802905] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2007.837350] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2010.825738] Lustre: lustre-MDD0000: changelog on [ 2011.697917] Lustre: lustre-MDD0001: changelog on [ 2358.192289] Lustre: lustre-MDD0001: changelog off [ 2358.826649] Lustre: 130319:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2359.130454] Lustre: lustre-MDD0000: changelog off [ 2362.851035] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2362.853076] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2362.853469] LustreError: Skipped 1 previous similar message [ 2362.857700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2362.860072] Lustre: Skipped 2 previous similar messages [ 2366.749646] Lustre: server umount lustre-OST0000 complete [ 2368.992183] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2382.303104] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2382.398926] Lustre: server umount lustre-MDT0000 complete [ 2383.328255] LustreError: 131378:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2383.333184] LustreError: 131378:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2383.614431] LustreError: 131088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772169658 with bad export cookie 2990742574374258558 [ 2383.617800] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2383.618640] LustreError: 131088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2383.734566] Lustre: server umount lustre-MDT0001 complete [ 2387.215557] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 2388.072152] Key type lgssc unregistered [ 2388.182359] LNet: 140346:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2388.184674] LNetError: 140346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2388.195259] LNet: Removed LNI 192.168.204.128@tcp [ 2388.449105] Key type .llcrypt unregistered [ 2388.450110] Key type ._llcrypt unregistered [ 2394.637377] Key type ._llcrypt registered [ 2394.638380] Key type .llcrypt registered [ 2394.675473] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 2400.880406] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 2401.161460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2401.207478] alg: No test for adler32 (adler32-zlib) [ 2402.058929] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 2402.140179] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 2403.719130] Key type lgssc registered [ 2404.047636] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2415.188757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2415.195349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2416.296264] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2416.306716] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2416.341272] Lustre: lustre-MDT0000: new disk, initializing [ 2416.365826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2416.372689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2417.725777] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2421.962551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2422.001133] Lustre: 143485:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2422.023292] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2422.025644] Lustre: Skipped 1 previous similar message [ 2422.063707] Lustre: lustre-MDT0001: new disk, initializing [ 2422.094443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2422.104772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2422.107741] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2423.303784] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2426.604083] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2427.758174] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2429.980633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2430.079441] Lustre: lustre-OST0000: new disk, initializing [ 2430.081313] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2430.108508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2431.913599] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2432.822700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2432.826119] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2432.855286] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2434.936375] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2437.163590] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_module obdecho/obdecho [ 2437.768533] LustreError: 146097:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 2437.771552] LustreError: 146097:0:(obd_config.c:880:class_setup()) setup ec failed (-22) [ 2438.343117] LustreError: 146193:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 2442.719818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2442.722291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2442.726397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2443.232360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2443.233813] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2443.235736] Lustre: Skipped 1 previous similar message [ 2443.239724] Lustre: Skipped 1 previous similar message [ 2447.706915] Lustre: server umount lustre-MDT0000 complete [ 2448.921181] LustreError: 143476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772169723 with bad export cookie 573965437541317797 [ 2448.923922] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2448.925821] LustreError: 143476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2449.036305] Lustre: server umount lustre-MDT0001 complete [ 2459.560294] Lustre: server umount lustre-OST0000 complete [ 2466.588196] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 2470.225860] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 2482.061745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2482.151782] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2482.163087] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2482.203710] Lustre: lustre-MDT0000: new disk, initializing [ 2482.228877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2482.235910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2483.551060] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2488.291935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2488.345452] Lustre: 149867:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2488.378411] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2488.381582] Lustre: Skipped 1 previous similar message [ 2488.427779] Lustre: lustre-MDT0001: new disk, initializing [ 2488.464817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2488.478569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2488.482034] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2489.737861] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2492.802578] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2494.038775] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2496.341983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2496.441549] Lustre: lustre-OST0000: new disk, initializing [ 2496.443582] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2496.469681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2498.251123] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2501.355034] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2502.913068] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2503.685768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2503.689185] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2503.744215] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2504.081934] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2505.702167] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2505.767045] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2508.771513] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2508.774616] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2508.776388] LustreError: Skipped 1 previous similar message [ 2508.785464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2508.791433] Lustre: Skipped 1 previous similar message [ 2512.528302] Lustre: server umount lustre-OST0000 complete [ 2513.888889] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2513.889323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2513.893821] Lustre: Skipped 2 previous similar messages [ 2519.009367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2519.013123] Lustre: Skipped 2 previous similar messages [ 2528.223128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2528.303863] Lustre: server umount lustre-MDT0000 complete [ 2529.249001] LustreError: 149877:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2529.255847] LustreError: 149877:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2529.595740] LustreError: 150645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772169804 with bad export cookie 573965437541318742 [ 2529.601324] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2529.719932] Lustre: server umount lustre-MDT0001 complete [ 2531.003375] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 00:23:24 (1772169804) [ 2531.514967] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 2532.056744] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 00:23:25 (1772169805) [ 2534.558416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2534.730339] LustreError: 153875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2534.757331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2536.084510] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2538.861764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2540.513821] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2542.535898] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2543.848000] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2546.529025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2548.687380] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2550.492088] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2564.575971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2564.579337] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2564.584630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2564.587582] Lustre: Skipped 2 previous similar messages [ 2568.989247] Lustre: server umount lustre-MDT0000 complete [ 2570.170178] LustreError: 153854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772169844 with bad export cookie 573965437541319624 [ 2570.173860] LustreError: 153854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2570.174058] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2570.207872] LustreError: 155134:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2575.327830] LustreError: 153869:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2575.335184] LustreError: 153869:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2576.395643] Lustre: server umount lustre-MDT0001 complete [ 2578.614863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2582.797363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2582.875917] LustreError: 157397:0:(llog.c:1643:llog_backup()) MGC192.168.204.128@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2582.878492] Lustre: 157397:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.204.128@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2591.838217] LustreError: 157404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2599.904248] LustreError: 157421:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2599.910278] LustreError: 157421:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2617.438537] LustreError: 157404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2617.443124] LustreError: 157404:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 2643.551331] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2643.553066] Lustre: 157397:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.204.128@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2650.080827] LustreError: 157421:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2650.091563] LustreError: 157421:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 22 previous similar messages [ 2673.631346] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2673.633629] Lustre: 153866:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.204.128@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2684.391821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2684.395138] Lustre: Skipped 2 previous similar messages [ 2685.803135] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2688.920302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2689.507614] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2689.509953] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7f7228642009cfe [ 2690.834359] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2692.517731] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2694.295623] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [ 2694.635591] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2694.640047] Lustre: Skipped 1 previous similar message [ 2695.527537] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2699.113731] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2709.986809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2709.988498] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2709.995322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2710.002029] Lustre: Skipped 5 previous similar messages [ 2710.005451] Lustre: Skipped 8 previous similar messages [ 2710.133495] Lustre: server umount lustre-MDT0000 complete [ 2711.544280] LustreError: 153854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772169986 with bad export cookie 573965437541326078 [ 2711.547851] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2711.550156] LustreError: 153854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2711.693173] Lustre: server umount lustre-MDT0001 complete [ 2723.527552] Lustre: server umount lustre-OST0000 complete [ 2732.089912] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 2736.728169] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 2753.167652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2753.308848] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2753.325438] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2753.380743] Lustre: lustre-MDT0000: new disk, initializing [ 2753.417081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2753.421237] Lustre: Skipped 1 previous similar message [ 2753.431064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2755.149281] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2760.311128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2760.371389] Lustre: 162215:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2760.514984] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2762.043421] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2765.951207] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2767.765965] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2771.171900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2772.663268] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2772.699476] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2773.590987] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2777.584284] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2779.391545] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2779.485171] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2781.187109] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2781.312768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2783.204213] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2783.205436] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2783.210167] LustreError: Skipped 1 previous similar message [ 2783.221225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2783.227698] Lustre: Skipped 2 previous similar messages [ 2788.389859] Lustre: server umount lustre-OST0000 complete [ 2791.391877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2796.157529] Lustre: server umount lustre-MDT0000 complete [ 2797.774357] LustreError: 162993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772170072 with bad export cookie 573965437541342878 [ 2797.777535] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2797.778466] LustreError: 162993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2797.901399] Lustre: server umount lustre-MDT0001 complete [ 2801.318739] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 00:27:54 (1772170074) [ 2803.573918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2803.776211] LustreError: 166221:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2803.784539] LustreError: 166221:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 26 previous similar messages [ 2805.461324] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2809.470995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2811.160655] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2813.602177] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2815.304293] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2818.466272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2821.174171] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2823.327032] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2833.514417] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 00:28:26 (1772170106) [ 2834.912302] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2834.916611] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2834.923169] Lustre: Skipped 3 previous similar messages [ 2840.810683] Lustre: server umount lustre-OST0000 complete [ 2848.679112] Lustre: server umount lustre-MDT0000 complete [ 2850.069249] LustreError: 166200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772170124 with bad export cookie 573965437541343746 [ 2850.071579] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2850.073382] LustreError: 166200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2850.187711] Lustre: server umount lustre-MDT0001 complete [ 2855.086786] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 2856.533495] Key type lgssc unregistered [ 2856.721579] LNet: 169594:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2856.727133] LNetError: 169594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2856.735781] LNet: Removed LNI 192.168.204.128@tcp [ 2857.252574] Key type .llcrypt unregistered [ 2857.255471] Key type ._llcrypt unregistered [ 2866.439367] Key type ._llcrypt registered [ 2866.441542] Key type .llcrypt registered [ 2866.495261] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 2866.826932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2866.878100] alg: No test for adler32 (adler32-zlib) [ 2867.737167] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 2867.827389] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 2869.407216] Key type lgssc registered [ 2870.039473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2873.729451] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2873.741600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2889.183503] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2894.303310] LustreError: 170803:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.128@tcp: failed processing log, type 4: rc = -110 [ 2899.423692] LustreError: 170803:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2899.432995] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 2899.437792] LustreError: MGC192.168.204.128@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 2899.445734] LustreError: 170803:0:(tgt_mount.c:1822:server_start_targets()) failed to start server lustre-OST0000: -22 [ 2899.452322] LustreError: 170803:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -22 [ 2899.457858] LustreError: 170803:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 2899.497243] Lustre: server umount lustre-OST0000 complete [ 2899.500406] LustreError: 170803:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 2903.412939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2904.630273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2905.887231] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2910.276267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2910.488501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2912.161474] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2913.851197] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2915.635309] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2918.239625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2918.346663] LustreError: 172891:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2918.353968] Lustre: 172891:0:(mgc_request_server.c:743:mgc_llog_local_copy()) MGC192.168.204.128@tcp: invalid local config log lustre-OST0000: rc = -22 [ 2918.432218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2920.992369] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2923.384226] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2925.541250] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2925.544038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2925.549491] Lustre: Skipped 2 previous similar messages [ 2925.553357] Lustre: Skipped 1 previous similar message [ 2930.411866] Lustre: server umount lustre-MDT0000 complete [ 2930.658430] LustreError: 171301:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2930.670107] LustreError: 171301:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2932.021248] LustreError: 171326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772170206 with bad export cookie 15767142972652264623 [ 2932.024542] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2932.026135] LustreError: 171326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2932.135406] Lustre: server umount lustre-MDT0001 complete [ 2943.915985] Lustre: server umount lustre-OST0000 complete [ 2947.258470] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 2951.946355] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 2969.622070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2969.769339] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2969.790065] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2969.836477] Lustre: lustre-MDT0000: new disk, initializing [ 2969.875084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2969.885168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2971.660776] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2977.053504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2977.106243] Lustre: 176263:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2977.134151] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2977.136222] Lustre: Skipped 1 previous similar message [ 2977.172268] Lustre: lustre-MDT0001: new disk, initializing [ 2977.205941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2977.219354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2977.222779] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2978.940178] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2982.932843] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2983.951112] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2986.719713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2986.855503] Lustre: lustre-OST0000: new disk, initializing [ 2986.859738] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2986.908649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2989.536573] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2992.937510] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2994.838538] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2995.209729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2995.217465] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2995.272687] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2995.988631] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2997.744729] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2997.822662] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3000.289775] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3000.293089] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3000.295201] LustreError: Skipped 1 previous similar message [ 3000.304714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3000.310968] Lustre: Skipped 2 previous similar messages [ 3004.816885] Lustre: server umount lustre-OST0000 complete [ 3007.968420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3007.972748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3007.979697] Lustre: Skipped 1 previous similar message [ 3007.982891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3010.529566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3010.530110] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3012.196510] Lustre: server umount lustre-MDT0000 complete [ 3013.591435] LustreError: 176255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772170288 with bad export cookie 15767142972652265309 [ 3013.597379] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3013.720396] Lustre: server umount lustre-MDT0001 complete [ 3016.922858] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 00:31:30 (1772170290) [ 3023.296088] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 3024.635923] Key type lgssc unregistered [ 3024.759333] LNet: 180790:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3024.763157] LNetError: 180790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3024.771291] LNet: Removed LNI 192.168.204.128@tcp [ 3025.093903] Key type .llcrypt unregistered [ 3025.094936] Key type ._llcrypt unregistered [ 3033.334746] Key type ._llcrypt registered [ 3033.336265] Key type .llcrypt registered [ 3033.387081] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 3033.793156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3033.799545] alg: No test for adler32 (adler32-zlib) [ 3034.688005] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 3034.780740] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 3036.383220] Key type lgssc registered [ 3036.757549] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3039.999912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3040.013554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3041.227628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3042.726536] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3045.779634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3045.924534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3047.693977] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3050.001709] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3051.755340] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3055.339775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3055.447690] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3055.449077] LustreError: 183605:0:(llog.c:1684:llog_backup()) MGC192.168.204.128@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3055.451940] Lustre: 183605:0:(mgc_request_server.c:740:mgc_llog_local_copy()) MGC192.168.204.128@tcp: can't backup local config lustre-OST0000: rc = -28 [ 3055.501871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3057.762524] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3060.259621] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3061.216383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3061.222149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3061.231596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3061.734102] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3061.736149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3061.742317] Lustre: Skipped 1 previous similar message [ 3066.850148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3066.854504] Lustre: Skipped 2 previous similar messages [ 3067.245439] Lustre: server umount lustre-MDT0000 complete [ 3068.973117] LustreError: 181950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772170343 with bad export cookie 12432859779157778344 [ 3068.978781] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3068.981915] LustreError: 181950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3069.125258] Lustre: server umount lustre-MDT0001 complete [ 3080.949236] Lustre: server umount lustre-OST0000 complete [ 3085.051936] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 3088.554737] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 3104.512430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3104.647675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3104.663233] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3104.710791] Lustre: lustre-MDT0000: new disk, initializing [ 3104.739579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3104.748804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3106.533565] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3112.186123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3112.247232] Lustre: 186980:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3112.308816] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3112.311320] Lustre: Skipped 1 previous similar message [ 3112.314183] LustreError: 187746:0:(llog.c:1684:llog_backup()) MGC192.168.204.128@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 3112.314825] LustreError: 186991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3112.319525] LustreError: 187746:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 3112.319590] Lustre: 187746:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.204.128@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 3112.328321] LustreError: 186991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3112.339178] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3112.343976] Lustre: Skipped 1 previous similar message [ 3112.384184] Lustre: lustre-MDT0001: new disk, initializing [ 3112.442362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3112.464598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3112.470421] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3114.199840] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3117.950070] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3119.221884] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3121.997426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3122.148205] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3122.150883] Lustre: Skipped 1 previous similar message [ 3122.154229] LustreError: 188700:0:(llog.c:1684:llog_backup()) MGC192.168.204.128@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3122.160517] LustreError: 188700:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 3122.165061] Lustre: 188700:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.204.128@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 3122.172845] Lustre: 188700:0:(mgc_request_server.c:763:mgc_llog_local_copy()) Skipped 1 previous similar message [ 3122.184381] Lustre: lustre-OST0000: new disk, initializing [ 3122.186795] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3122.221932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3124.516380] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3126.203831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3126.207084] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3126.240951] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3127.700501] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3129.427275] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3129.548753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3131.298946] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3131.384177] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3136.482703] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3136.483014] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3136.485687] LustreError: Skipped 1 previous similar message [ 3136.490802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3136.496159] Lustre: Skipped 1 previous similar message [ 3138.479555] Lustre: server umount lustre-OST0000 complete [ 3141.602485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3141.603725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3141.610375] Lustre: Skipped 2 previous similar messages [ 3146.480795] Lustre: server umount lustre-MDT0000 complete [ 3146.721971] LustreError: 186991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3146.734168] LustreError: 186991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3147.869229] LustreError: 187760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772170422 with bad export cookie 12432859779157779051 [ 3147.872401] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3147.872841] LustreError: 187760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3147.975768] Lustre: server umount lustre-MDT0001 complete [ 3152.518920] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 00:33:45 (1772170425) [ 3160.512337] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 3164.896720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3165.073204] LustreError: 191681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3165.090279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3166.906381] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3170.273511] LustreError: 191681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3170.824047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3172.724626] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3174.188812] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3177.658357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3180.707087] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3184.402109] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3184.524421] Lustre: lustre-OST0001: new disk, initializing [ 3184.527055] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3186.080536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3186.087375] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3186.141727] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3186.994334] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3193.468624] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3199.396096] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3205.405033] Lustre: DEBUG MARKER: ADD OST3 [ 3207.944620] loop: module loaded [ 3207.979896] loop0: detected capacity change from 0 to 204800000 [ 3208.048758] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3208.052335] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3208.056542] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3211.055804] loop0: detected capacity change from 0 to 204800000 [ 3213.442501] Lustre: DEBUG MARKER: STOP OST3 [ 3213.664826] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3213.750994] Lustre: lustre-OST0003: new disk, initializing [ 3213.752909] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 3213.792326] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 3213.794516] Lustre: Skipped 3 previous similar messages [ 3214.329826] Lustre: Failing over lustre-OST0003 [ 3214.383204] Lustre: server umount lustre-OST0003 complete [ 3215.178963] LustreError: 193132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3216.203345] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3221.180416] Lustre: DEBUG MARKER: START OST3 again [ 3224.488400] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3226.616613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 3226.621068] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 3226.636236] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 3227.338470] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3241.784064] Lustre: Failing over lustre-OST0003 [ 3241.845187] Lustre: server umount lustre-OST0003 complete [ 3241.953988] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3241.956174] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3241.958220] LustreError: Skipped 1 previous similar message [ 3241.966227] LustreError: 196059:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3241.975988] LustreError: 196059:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 3255.266191] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3255.268861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3255.269637] Lustre: Skipped 3 previous similar messages [ 3255.271388] Lustre: Skipped 2 previous similar messages [ 3260.391854] LustreError: 193130:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3260.398619] LustreError: 193130:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3261.177465] Lustre: server umount lustre-MDT0000 complete [ 3262.848692] LustreError: 194693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772170537 with bad export cookie 12432859779157779933 [ 3262.854559] LustreError: 194693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3262.854966] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3272.673692] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3272.677619] Lustre: Skipped 11 previous similar messages [ 3277.279154] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3277.412837] Lustre: server umount lustre-MDT0001 complete [ 3279.402783] Lustre: server umount lustre-OST0000 complete [ 3281.396376] Lustre: server umount lustre-OST0001 complete [ 3284.428330] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 3287.816345] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 3304.986078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3305.115371] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3305.129206] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3305.173659] Lustre: lustre-MDT0000: new disk, initializing [ 3305.205267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3305.208109] Lustre: Skipped 1 previous similar message [ 3305.216195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3306.878844] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3312.245282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3312.300324] Lustre: 201154:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3312.431156] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3314.113731] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3317.750514] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3319.377217] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3322.689647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3322.869300] Lustre: lustre-OST0000: new disk, initializing [ 3322.872471] Lustre: Skipped 1 previous similar message [ 3322.876606] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3322.878198] Lustre: Skipped 2 previous similar messages [ 3324.110968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3324.113185] Lustre: Skipped 1 previous similar message [ 3324.114791] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3324.169429] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3325.277440] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3328.935289] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3330.796517] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3330.888705] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3332.677788] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3332.809248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3334.630901] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3334.632490] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3334.633466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3334.634622] Lustre: Skipped 4 previous similar messages [ 3334.639160] LustreError: Skipped 1 previous similar message [ 3339.939633] Lustre: server umount lustre-OST0000 complete [ 3343.328479] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3347.842932] Lustre: server umount lustre-MDT0000 complete [ 3349.712770] LustreError: 201147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772170624 with bad export cookie 12432859779157782180 [ 3349.718464] LustreError: 201147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3349.718817] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3349.922401] Lustre: server umount lustre-MDT0001 complete [ 3351.567370] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 00:37:04 (1772170624) [ 3359.020071] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 3363.057483] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 3378.365458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3378.526166] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3378.550489] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3378.606541] Lustre: lustre-MDT0000: new disk, initializing [ 3378.644858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3380.268857] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3385.855602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3385.929774] Lustre: 207316:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3386.094737] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3387.891927] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3391.826838] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3393.167418] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3396.432472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3397.999588] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3399.213084] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3403.416769] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3404.892238] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3404.997326] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3406.881888] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3406.996816] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3408.358091] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3408.358657] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3408.361524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3408.361537] Lustre: Skipped 6 previous similar messages [ 3408.365842] Lustre: Skipped 3 previous similar messages [ 3408.381038] LustreError: Skipped 1 previous similar message [ 3422.175152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3422.263420] Lustre: server umount lustre-OST0000 complete [ 3427.296746] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3430.233671] Lustre: server umount lustre-MDT0000 complete [ 3431.867288] LustreError: 207308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772170706 with bad export cookie 12432859779157783062 [ 3431.873193] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3431.874517] LustreError: 207308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3432.028904] Lustre: server umount lustre-MDT0001 complete [ 3436.304434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3436.526915] LustreError: 211124:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3436.534895] LustreError: 211124:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 3436.563759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3436.567524] Lustre: Skipped 5 previous similar messages [ 3438.603734] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3442.899145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3444.772395] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3447.274505] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3448.755953] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3452.270754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3455.115931] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3457.900554] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3471.844383] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3476.585346] Lustre: server umount lustre-OST0000 complete [ 3479.008272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3479.015916] Lustre: Skipped 4 previous similar messages [ 3484.331462] Lustre: server umount lustre-MDT0000 complete [ 3486.094225] LustreError: 211105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772170760 with bad export cookie 12432859779157783944 [ 3486.101382] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3486.103954] LustreError: 211105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3486.246626] Lustre: server umount lustre-MDT0001 complete [ 3592.428877] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 3593.775804] Key type lgssc unregistered [ 3593.923571] LNet: 214450:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3593.928185] LNetError: 214450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3593.941522] LNet: Removed LNI 192.168.204.128@tcp [ 3594.399150] Key type .llcrypt unregistered [ 3594.401748] Key type ._llcrypt unregistered [ 3604.877124] Key type ._llcrypt registered [ 3604.878776] Key type .llcrypt registered [ 3604.942043] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 3612.436239] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 3612.745270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3612.761087] alg: No test for adler32 (adler32-zlib) [ 3613.601215] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 3613.675093] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 3615.255220] Key type lgssc registered [ 3615.855590] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3633.092857] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3633.108715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3634.231638] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3634.248407] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3634.290202] Lustre: lustre-MDT0000: new disk, initializing [ 3634.314603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3634.320651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3635.853505] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3641.841722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3641.909308] Lustre: 217743:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3641.957746] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3641.962142] Lustre: Skipped 1 previous similar message [ 3642.009953] Lustre: lustre-MDT0001: new disk, initializing [ 3642.063761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3642.085680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3642.093838] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3643.995330] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3648.048644] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3649.732434] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3653.062468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3653.183092] Lustre: lustre-OST0000: new disk, initializing [ 3653.184910] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3653.210957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3655.806569] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3660.109192] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3660.290755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3660.296553] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3660.347819] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3662.168529] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3662.295323] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3664.073117] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3664.165153] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3665.382877] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3665.382996] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3665.392235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3665.395600] LustreError: Skipped 1 previous similar message [ 3665.399806] Lustre: Skipped 1 previous similar message [ 3667.221749] Lustre: server umount lustre-OST0000 complete [ 3670.498420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3670.499699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3670.507121] Lustre: Skipped 2 previous similar messages [ 3670.514879] Lustre: Skipped 1 previous similar message [ 3675.617055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3675.620634] Lustre: Skipped 1 previous similar message [ 3680.737300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3680.741317] Lustre: Skipped 1 previous similar message [ 3683.295199] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3683.371261] Lustre: server umount lustre-MDT0000 complete [ 3685.149984] LustreError: 217733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772170959 with bad export cookie 15291690710089809933 [ 3685.155520] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3685.157817] LustreError: 217733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3685.317445] Lustre: server umount lustre-MDT0001 complete [ 3694.390586] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 3699.317069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3699.501665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3701.296750] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3704.801643] LustreError: 222255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3704.813154] LustreError: 222255:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3705.210285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3705.355513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3707.065159] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3708.468903] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3711.812255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3714.485932] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3718.774215] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3718.866816] Lustre: lustre-OST0001: new disk, initializing [ 3718.870942] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3718.930241] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3718.933932] Lustre: Skipped 1 previous similar message [ 3720.676938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3720.683762] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3720.741116] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3721.570235] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3727.861947] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3733.339552] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3744.064800] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 00:43:37 (1772171017) [ 3745.251382] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3745.259216] Lustre: Skipped 2 previous similar messages [ 3745.260851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3745.265887] Lustre: Skipped 3 previous similar messages [ 3747.272571] Lustre: server umount lustre-MDT0000 complete [ 3749.144647] LustreError: 222234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772171023 with bad export cookie 15291690710089810801 [ 3749.152729] LustreError: 222234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3749.153268] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3749.324312] Lustre: server umount lustre-MDT0001 complete [ 3761.491442] Lustre: server umount lustre-OST0000 complete [ 3773.560416] Lustre: server umount lustre-OST0001 complete [ 3776.098248] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 3780.484902] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 3796.859051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3796.988162] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3797.004354] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3797.048821] Lustre: lustre-MDT0000: new disk, initializing [ 3797.076402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3797.086690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3798.867729] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3804.645872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3804.722252] Lustre: 228401:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3804.883993] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3806.447693] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3810.496133] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3812.274203] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3815.702933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3815.879655] Lustre: lustre-OST0000: new disk, initializing [ 3815.882715] Lustre: Skipped 1 previous similar message [ 3815.886420] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3815.890219] Lustre: Skipped 2 previous similar messages [ 3817.170456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3817.178281] Lustre: Skipped 1 previous similar message [ 3817.182820] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3817.244245] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3818.541668] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3822.202850] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3823.865919] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3824.002791] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3826.034699] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3826.175304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3827.682993] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3827.683239] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3827.687341] LustreError: Skipped 1 previous similar message [ 3827.696453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3827.701747] Lustre: Skipped 1 previous similar message [ 3833.319154] Lustre: server umount lustre-OST0000 complete [ 3835.872419] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3835.877871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3835.885898] Lustre: Skipped 1 previous similar message [ 3841.270772] Lustre: server umount lustre-MDT0000 complete [ 3843.041146] LustreError: 228412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3843.053357] LustreError: 228412:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3843.156952] LustreError: 229180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772171117 with bad export cookie 15291690710089812929 [ 3843.163855] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3843.164128] LustreError: 229180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3843.311979] Lustre: server umount lustre-MDT0001 complete [ 3847.411330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3847.654678] LustreError: 232210:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3847.682165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3847.684859] Lustre: Skipped 2 previous similar messages [ 3849.291566] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3852.770231] LustreError: 232210:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3853.552288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3855.372665] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3857.634527] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3859.250857] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3861.896703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3864.260316] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3866.949484] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4011.493380] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4011.496136] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4011.499448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4011.507052] Lustre: Skipped 2 previous similar messages [ 4011.511948] Lustre: Skipped 6 previous similar messages [ 4017.440593] Lustre: server umount lustre-OST0000 complete [ 4025.472755] Lustre: server umount lustre-MDT0000 complete [ 4026.849407] LustreError: 232210:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4026.857114] LustreError: 232210:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4027.267191] LustreError: 232191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772171301 with bad export cookie 15291690710089813811 [ 4027.272763] LustreError: 232191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4027.274953] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4027.408419] Lustre: server umount lustre-MDT0001 complete [ 4136.559856] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 4138.120651] Key type lgssc unregistered [ 4138.288673] LNet: 235466:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4138.294986] LNetError: 235466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4138.311756] LNet: Removed LNI 192.168.204.128@tcp [ 4138.714112] Key type .llcrypt unregistered [ 4138.715178] Key type ._llcrypt unregistered [ 4149.409259] Key type ._llcrypt registered [ 4149.410327] Key type .llcrypt registered [ 4149.449240] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 4157.748875] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 4158.211496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4158.245724] alg: No test for adler32 (adler32-zlib) [ 4159.125524] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 4159.232814] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 4160.855339] Key type lgssc registered [ 4161.202012] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4180.299920] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 4184.630762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4184.639733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4185.751594] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4185.773044] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4185.828846] Lustre: lustre-MDT0000: new disk, initializing [ 4185.860881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4185.871616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4187.488482] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4193.100265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4193.204127] Lustre: 239459:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4193.255460] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4193.259293] Lustre: Skipped 1 previous similar message [ 4193.306510] Lustre: lustre-MDT0001: new disk, initializing [ 4193.366614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4193.391712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4193.398867] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4195.034908] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4197.695382] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4201.122087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4201.246250] Lustre: lustre-OST0000: new disk, initializing [ 4201.249247] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4201.291486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4204.004023] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4205.209530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4205.215654] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4205.286893] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4209.761804] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4209.847798] Lustre: lustre-OST0001: new disk, initializing [ 4209.849590] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4209.898369] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4212.255961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4212.260966] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4212.309214] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4212.708582] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4219.329981] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4225.953762] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4237.798840] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4237.804976] Lustre: Skipped 2 previous similar messages [ 4237.805238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4237.812072] Lustre: Skipped 3 previous similar messages [ 4240.729735] Lustre: server umount lustre-MDT0000 complete [ 4242.254264] LustreError: 242722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772171516 with bad export cookie 11519637167840155925 [ 4242.257921] LustreError: 242722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4242.258357] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4242.381458] Lustre: server umount lustre-MDT0001 complete [ 4254.163956] Lustre: server umount lustre-OST0000 complete [ 4266.240812] Lustre: server umount lustre-OST0001 complete [ 4271.769854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4275.551447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4279.605301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4284.072702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4293.146995] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 4298.051524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4298.065067] Lustre: lustre-MDT0000: reset Object Index mappings [ 4298.343978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4300.077520] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4303.482824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4303.495889] Lustre: lustre-MDT0001: reset Object Index mappings [ 4305.447307] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4306.707831] Lustre: 247991:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4310.132363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4310.326138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4310.330546] Lustre: Skipped 1 previous similar message [ 4312.935411] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4314.404527] LustreError: 248853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4314.410918] LustreError: 248853:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4317.071838] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4319.417899] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4322.729556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4322.730045] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 4329.247455] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4330.850496] Lustre: 249837:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4338.147523] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4338.153478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4338.156477] Lustre: Skipped 3 previous similar messages [ 4338.163906] Lustre: Skipped 3 previous similar messages [ 4343.266504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4343.270540] Lustre: Skipped 3 previous similar messages [ 4343.735344] Lustre: server umount lustre-MDT0000 complete [ 4345.519794] LustreError: 249839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772171620 with bad export cookie 11519637167840162715 [ 4345.526294] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4345.527222] LustreError: 249839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4345.690344] Lustre: server umount lustre-MDT0001 complete [ 4357.586679] Lustre: server umount lustre-OST0000 complete [ 4368.648356] Lustre: server umount lustre-OST0001 complete [ 4378.028404] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 00:54:11 (1772171651) [ 4385.344195] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 4390.166676] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 4408.138779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4408.280487] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4408.296290] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4408.345946] Lustre: lustre-MDT0000: new disk, initializing [ 4408.378864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4408.381623] Lustre: Skipped 1 previous similar message [ 4408.390195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4410.405232] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4416.319297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4416.377901] Lustre: 254053:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4416.420817] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4416.425952] Lustre: Skipped 1 previous similar message [ 4416.476042] Lustre: lustre-MDT0001: new disk, initializing [ 4416.549060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4416.555874] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4418.383711] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4422.491640] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4424.256460] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4427.697515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4429.198387] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4429.260934] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4430.534364] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4434.269493] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4435.817976] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4435.946977] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4437.793243] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4437.899838] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4439.522898] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4439.524964] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4439.529420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4439.536045] Lustre: Skipped 1 previous similar message [ 4439.543625] Lustre: Skipped 1 previous similar message [ 4444.643303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4444.647335] Lustre: Skipped 1 previous similar message [ 4453.343166] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4453.432464] Lustre: server umount lustre-OST0000 complete [ 4457.440912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4457.446752] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4457.455989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4457.462800] Lustre: Skipped 2 previous similar messages [ 4461.251930] Lustre: server umount lustre-MDT0000 complete [ 4463.126182] LustreError: 257187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772171737 with bad export cookie 11519637167840164808 [ 4463.133714] LustreError: 257187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4463.135691] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4463.304055] Lustre: server umount lustre-MDT0001 complete [ 4472.005538] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 4476.360239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4476.540358] LustreError: 258564:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4476.556285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4476.559594] Lustre: Skipped 2 previous similar messages [ 4478.191183] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4482.017312] LustreError: 258564:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4482.290659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4484.357543] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4485.797428] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4488.634237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4491.209392] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4495.232799] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4495.307979] Lustre: lustre-OST0001: new disk, initializing [ 4495.309127] Lustre: Skipped 1 previous similar message [ 4495.310629] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4495.312163] Lustre: Skipped 1 previous similar message [ 4497.299143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4497.302407] Lustre: Skipped 1 previous similar message [ 4497.306047] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4497.349492] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4497.837724] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4503.546409] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4505.235618] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4512.326699] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4512.332262] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4517.862898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4517.867955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4517.871571] Lustre: Skipped 4 previous similar messages [ 4517.875584] Lustre: Skipped 4 previous similar messages [ 4520.827111] Lustre: server umount lustre-MDT0000 complete [ 4522.657336] LustreError: 259281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772171797 with bad export cookie 11519637167840165683 [ 4522.664771] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4522.664849] LustreError: 259281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4522.837245] Lustre: server umount lustre-MDT0001 complete [ 4535.075671] Lustre: server umount lustre-OST0000 complete [ 4547.288628] Lustre: server umount lustre-OST0001 complete [ 4553.904118] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 4558.707048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4558.942481] LustreError: 263842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4558.978713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4558.983289] Lustre: Skipped 3 previous similar messages [ 4560.909490] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4564.449810] LustreError: 263842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4565.161956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4566.930973] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4568.436748] Lustre: 264947:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4571.391215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4573.762917] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4574.567119] LustreError: 265302:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4577.253767] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4580.240123] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4584.059032] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 4584.074338] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 4589.661087] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4590.924476] Lustre: 266803:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4598.342734] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 4598.348232] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 4602.132899] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 00:57:55 (1772171875) [ 4615.140699] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4615.146635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4615.149580] Lustre: Skipped 2 previous similar messages [ 4615.153895] Lustre: Skipped 2 previous similar messages [ 4620.211936] Lustre: server umount lustre-MDT0000 complete [ 4620.258772] LustreError: 266560:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4620.264044] LustreError: 266560:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4621.705870] LustreError: 263821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772171896 with bad export cookie 11519637167840171990 [ 4621.712032] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4621.713486] LustreError: 263821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4636.127184] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4636.247143] Lustre: server umount lustre-MDT0001 complete [ 4644.237697] Lustre: server umount lustre-OST0000 complete [ 4652.220906] Lustre: server umount lustre-OST0001 complete [ 4655.017957] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 4659.566340] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 4681.675849] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 4685.857122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4685.995196] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4686.009765] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4686.049619] Lustre: lustre-MDT0000: new disk, initializing [ 4686.088274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4687.740389] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4693.028527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4693.118780] Lustre: 270988:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4693.264536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4693.269071] Lustre: Skipped 4 previous similar messages [ 4693.293318] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4695.055909] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4701.338385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4702.670654] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4704.331205] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4710.179586] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4712.349084] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4713.013330] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4718.186598] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4719.907682] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4719.913431] Lustre: Skipped 1 previous similar message [ 4729.924923] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 01:00:03 (1772172003) [ 4732.902614] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4732.905049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4732.910530] Lustre: Skipped 4 previous similar messages [ 4732.914526] Lustre: Skipped 7 previous similar messages [ 4736.985450] Lustre: server umount lustre-MDT0000 complete [ 4738.018798] LustreError: 271957:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4738.028800] LustreError: 271957:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 4738.605536] LustreError: 270979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772172013 with bad export cookie 11519637167840179858 [ 4738.611311] LustreError: 270979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4738.611317] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4738.742647] Lustre: server umount lustre-MDT0001 complete [ 4750.356333] Lustre: server umount lustre-OST0000 complete [ 4762.372276] Lustre: server umount lustre-OST0001 complete [ 4769.910344] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 4773.989825] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 4790.067151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4790.162287] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4790.175625] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4790.177624] Lustre: Skipped 4 previous similar messages [ 4790.212918] Lustre: lustre-MDT0000: new disk, initializing [ 4790.214296] Lustre: Skipped 3 previous similar messages [ 4790.242987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4790.245543] Lustre: Skipped 3 previous similar messages [ 4791.930335] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4797.188192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4797.250886] Lustre: 278035:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4797.392643] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4797.397569] Lustre: Skipped 2 previous similar messages [ 4799.132578] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4802.733092] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4804.199217] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4807.246382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4808.830507] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4809.774878] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4813.208945] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4814.193464] Lustre: Mounted lustre-client [ 4814.589131] LustreError: 280532:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff97513884d000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 4814.638312] Lustre: Unmounted lustre-client [ 4819.427926] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4819.429171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4819.442669] Lustre: Skipped 3 previous similar messages [ 4824.733011] Lustre: server umount lustre-OST0000 complete [ 4828.128495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4828.133239] LustreError: Skipped 1 previous similar message [ 4832.632650] Lustre: server umount lustre-MDT0000 complete [ 4834.509332] LustreError: 278025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772172109 with bad export cookie 11519637167840181384 [ 4834.517555] LustreError: 278025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4834.519365] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4834.719294] Lustre: server umount lustre-MDT0001 complete [ 4839.573521] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 4840.789190] Key type lgssc unregistered [ 4866.527419] LNet: 281809:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4866.531115] LNetError: 281809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4945.384091] LNet: Removed LNI 192.168.204.128@tcp [ 4946.258129] Key type .llcrypt unregistered [ 4946.259623] Key type ._llcrypt unregistered [ 4955.799307] Key type ._llcrypt registered [ 4955.800378] Key type .llcrypt registered [ 4955.843489] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 4962.715689] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 4963.163302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4963.207774] alg: No test for adler32 (adler32-zlib) [ 4964.096926] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 4964.194326] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 4965.783199] Key type lgssc registered [ 4966.198852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4975.890631] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 01:04:09 (1772172249) [ 4978.008371] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4978.013495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4979.097336] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4979.108339] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4979.147847] Lustre: lustre-MDT0000: new disk, initializing [ 4979.171223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4979.176869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4980.546272] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4989.935940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4989.984427] Lustre: 284971:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4990.014525] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4990.017112] Lustre: Skipped 1 previous similar message [ 4990.067298] Lustre: lustre-MDT0001: new disk, initializing [ 4990.104814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4990.124516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4990.128128] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4991.492654] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4994.666362] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4995.983443] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4998.310502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4998.407284] Lustre: lustre-OST0000: new disk, initializing [ 4998.409656] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4998.436180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4999.798963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4999.803407] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4999.837741] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5000.323361] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5003.364376] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5010.400933] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5010.401295] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5010.403517] LustreError: Skipped 1 previous similar message [ 5010.408296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5010.410877] Lustre: Skipped 1 previous similar message [ 5013.265465] Lustre: server umount lustre-OST0000 complete [ 5015.520453] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5015.521121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5015.525224] Lustre: Skipped 2 previous similar messages [ 5020.629139] Lustre: server umount lustre-MDT0000 complete [ 5021.876331] LustreError: 284960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772172296 with bad export cookie 5888263532025490610 [ 5021.879120] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5021.879559] LustreError: 284960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5022.001992] Lustre: server umount lustre-MDT0001 complete [ 5025.356855] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 5026.447377] Key type lgssc unregistered [ 5026.578385] LNet: 288338:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5026.580459] LNetError: 288338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5026.592243] LNet: Removed LNI 192.168.204.128@tcp [ 5026.872846] Key type .llcrypt unregistered [ 5026.873823] Key type ._llcrypt unregistered [ 5035.885773] Key type ._llcrypt registered [ 5035.886723] Key type .llcrypt registered [ 5035.924613] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 5036.238657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5036.265060] alg: No test for adler32 (adler32-zlib) [ 5037.118456] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 5037.218686] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 5038.799186] Key type lgssc registered [ 5039.170507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5041.930655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5041.938751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5043.128806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5044.593785] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5047.726698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5047.857733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5049.329768] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5051.276973] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5052.633427] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5055.099051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5055.213367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5057.260350] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5059.238086] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5072.907113] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5073.588208] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5075.047065] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.28@tcp (stopping) [ 5075.149915] Lustre: server umount lustre-MDT0000 complete [ 5076.535698] LustreError: 289693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772172351 with bad export cookie 4462983469666578506 [ 5076.539967] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5076.543085] LustreError: 289693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5079.008559] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5079.009170] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5079.012137] Lustre: Skipped 1 previous similar message [ 5079.016579] LustreError: 289709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5079.023148] LustreError: 289709:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5080.158423] LustreError: 289709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5080.165850] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.28@tcp (stopping) [ 5082.755474] Lustre: server umount lustre-MDT0001 complete [ 5085.286149] Lustre: lustre-OST0000: Not available for connect from 192.168.204.28@tcp (stopping) [ 5085.288500] Lustre: Skipped 1 previous similar message [ 5090.333085] Lustre: server umount lustre-OST0000 complete [ 5093.661226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5095.143093] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5096.281811] Lustre: 293464:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.128@tcp [ 5097.063045] Lustre: server umount MGS complete [ 5100.458130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5100.641879] LustreError: 294031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5100.651643] LustreError: 294031:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5100.711102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5102.203816] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5105.694049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5105.759341] LustreError: 294059:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5105.767265] LustreError: 294059:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5105.873097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5107.430472] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5110.886085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5113.191306] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5115.339767] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5126.386258] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 10 sec [ 5127.794920] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5128.489512] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5131.236967] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5131.240318] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5131.246341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5131.249719] Lustre: Skipped 1 previous similar message [ 5131.935865] Lustre: server umount lustre-OST0000 complete [ 5135.511298] Lustre: server umount lustre-MDT0000 complete [ 5136.352681] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5136.353583] LustreError: 294034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5136.357768] Lustre: Skipped 2 previous similar messages [ 5136.366454] LustreError: 294034:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5137.137765] LustreError: 294015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772172411 with bad export cookie 4462983469666579332 [ 5137.140436] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5137.141621] LustreError: 294015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5137.296777] Lustre: server umount lustre-MDT0001 complete [ 5141.656091] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 5142.805988] Key type lgssc unregistered [ 5142.936468] LNet: 297232:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5142.939751] LNetError: 297232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5142.951344] LNet: Removed LNI 192.168.204.128@tcp [ 5143.276647] Key type .llcrypt unregistered [ 5143.278321] Key type ._llcrypt unregistered [ 5150.324895] Key type ._llcrypt registered [ 5150.326472] Key type .llcrypt registered [ 5150.385603] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 5157.520421] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 5157.836768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5157.849683] alg: No test for adler32 (adler32-zlib) [ 5158.712871] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 5158.800806] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 5160.391165] Key type lgssc registered [ 5160.812806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5174.527303] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5174.536216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5175.630365] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5175.643370] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5175.683052] Lustre: lustre-MDT0000: new disk, initializing [ 5175.707871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5175.714517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5177.278326] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5182.249841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5182.300428] Lustre: 300329:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5182.327374] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5182.330193] Lustre: Skipped 1 previous similar message [ 5182.373104] Lustre: lustre-MDT0001: new disk, initializing [ 5182.410349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5182.427117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5182.431363] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5184.149526] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5187.782410] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5189.278828] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5192.274645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5192.406684] Lustre: lustre-OST0000: new disk, initializing [ 5192.409740] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5192.447316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5194.827224] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5197.828507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5197.832135] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5197.881122] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5198.566392] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5200.262730] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5200.369598] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5201.984067] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5202.066898] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5202.912485] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5202.913623] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5202.915937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5202.917199] LustreError: Skipped 1 previous similar message [ 5202.924853] Lustre: Skipped 1 previous similar message [ 5208.034555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5208.036614] Lustre: Skipped 2 previous similar messages [ 5213.153913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5213.156166] Lustre: Skipped 1 previous similar message [ 5217.247275] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5217.302995] Lustre: server umount lustre-OST0000 complete [ 5223.392679] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5223.392847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5223.394040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5223.397189] Lustre: Skipped 1 previous similar message [ 5228.512259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5228.516197] Lustre: Skipped 2 previous similar messages [ 5233.119188] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5233.194436] Lustre: server umount lustre-MDT0000 complete [ 5234.602313] LustreError: 301106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772172509 with bad export cookie 14583327369195927106 [ 5234.606273] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5234.607120] LustreError: 301106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5234.722084] Lustre: server umount lustre-MDT0001 complete [ 5238.321058] Lustre: DEBUG MARKER: == conf-sanity test 200a: check CPU partitions =========== 01:08:31 (1772172511) [ 5259.262314] Lustre: DEBUG MARKER: == conf-sanity test 200b: set CPU pattern using core selection ========================================================== 01:08:52 (1772172532) [ 5274.988460] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 5276.002333] Key type lgssc unregistered [ 5276.123335] LNet: 306871:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5276.125954] LNetError: 306871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5276.133275] LNet: Removed LNI 192.168.204.128@tcp [ 5276.432040] Key type .llcrypt unregistered [ 5276.433257] Key type ._llcrypt unregistered [ 5303.413826] Key type ._llcrypt registered [ 5303.415329] Key type .llcrypt registered [ 5303.451563] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 5303.550421] Key type .llcrypt unregistered [ 5303.551386] Key type ._llcrypt unregistered [ 5314.273061] Key type ._llcrypt registered [ 5314.274188] Key type .llcrypt registered [ 5314.312529] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 5314.409457] Key type .llcrypt unregistered [ 5314.410436] Key type ._llcrypt unregistered [ 5325.109677] Key type ._llcrypt registered [ 5325.110746] Key type .llcrypt registered [ 5325.147119] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 5325.246236] Key type .llcrypt unregistered [ 5325.247594] Key type ._llcrypt unregistered [ 5352.066364] Key type ._llcrypt registered [ 5352.067420] Key type .llcrypt registered [ 5352.102227] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 5352.196213] Key type .llcrypt unregistered [ 5352.197390] Key type ._llcrypt unregistered [ 5363.008901] Key type ._llcrypt registered [ 5363.010157] Key type .llcrypt registered [ 5363.049070] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 5363.149238] Key type .llcrypt unregistered [ 5363.150544] Key type ._llcrypt unregistered [ 5374.380102] Key type ._llcrypt registered [ 5374.381275] Key type .llcrypt registered [ 5374.416543] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 5374.513195] Key type .llcrypt unregistered [ 5374.514677] Key type ._llcrypt unregistered [ 5385.872392] Key type ._llcrypt registered [ 5385.873592] Key type .llcrypt registered [ 5385.915139] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 5386.016340] Key type .llcrypt unregistered [ 5386.017715] Key type ._llcrypt unregistered [ 5397.423247] Key type ._llcrypt registered [ 5397.424599] Key type .llcrypt registered [ 5397.466098] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 5397.570465] Key type .llcrypt unregistered [ 5397.571568] Key type ._llcrypt unregistered [ 5407.065399] hrtimer: interrupt took 4834953 ns [ 5420.801791] Key type ._llcrypt registered [ 5420.805737] Key type .llcrypt registered [ 5421.010493] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 5421.538426] Key type .llcrypt unregistered [ 5421.542205] Key type ._llcrypt unregistered [ 5489.887669] Key type ._llcrypt registered [ 5489.896906] Key type .llcrypt registered [ 5490.006386] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 5490.235420] Key type .llcrypt unregistered [ 5490.239882] Key type ._llcrypt unregistered [ 5513.330578] Key type ._llcrypt registered [ 5513.335964] Key type .llcrypt registered [ 5513.420575] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 5513.650550] Key type .llcrypt unregistered [ 5513.651637] Key type ._llcrypt unregistered [ 5537.192894] Key type ._llcrypt registered [ 5537.194532] Key type .llcrypt registered [ 5537.298530] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 5537.713279] Key type .llcrypt unregistered [ 5537.716835] Key type ._llcrypt unregistered [ 5587.272434] Key type ._llcrypt registered [ 5587.274353] Key type .llcrypt registered [ 5587.410879] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 5600.433269] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 5601.109589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5601.175728] alg: No test for adler32 (adler32-zlib) [ 5602.123294] Lustre: Lustre: Build Version: 2.17.50_152_g82f9dcf [ 5602.291576] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 5603.927408] Key type lgssc registered [ 5604.717770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5625.865426] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5497 sec ========== 01:14:58 (1772172898) [ 5626.989182] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:14:59 (1772172899) === [ 5629.140129] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:15:02 (1772172902) === [ 5660.573096] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 5662.979445] Key type lgssc unregistered [ 5663.198734] LNet: 325760:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5663.204033] LNetError: 325760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5663.234917] LNet: Removed LNI 192.168.204.128@tcp [ 5663.968152] Key type .llcrypt unregistered [ 5663.970444] Key type ._llcrypt unregistered