[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 575233514 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001017] APIC: Switch to symmetric I/O mode setup [ 0.003308] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008658] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010030] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011020] pid_max: default: 32768 minimum: 301 [ 0.012168] LSM: Security Framework initializing [ 0.013076] Yama: becoming mindful. [ 0.014065] SELinux: Initializing. [ 0.015109] *** VALIDATE selinux *** [ 0.023802] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028293] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030194] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032117] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033159] *** VALIDATE tmpfs *** [ 0.034549] *** VALIDATE proc *** [ 0.036235] *** VALIDATE cgroup *** [ 0.037013] *** VALIDATE cgroup2 *** [ 0.038311] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040187] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043036] Spectre V2 : User space: Vulnerable [ 0.045002] Speculative Store Bypass: Vulnerable [ 0.048218] debug: unmapping init [mem 0xffffffff90e59000-0xffffffff90e60fff] [ 0.050176] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.051599] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.052034] ... version: 2 [ 0.053011] ... bit width: 48 [ 0.054013] ... generic registers: 4 [ 0.055011] ... value mask: 0000ffffffffffff [ 0.056021] ... max period: 00007fffffffffff [ 0.057022] ... fixed-purpose events: 3 [ 0.058020] ... event mask: 000000070000000f [ 0.059411] rcu: Hierarchical SRCU implementation. [ 0.061808] smp: Bringing up secondary CPUs ... [ 0.062705] x86: Booting SMP configuration: [ 0.063041] .... node #0, CPUs: #1 #2 #3 [ 0.067036] smp: Brought up 1 node, 4 CPUs [ 0.069029] smpboot: Max logical packages: 1 [ 0.070024] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.231117] node 0 deferred pages initialised in 159ms [ 0.236024] devtmpfs: initialized [ 0.237300] x86/mm: Memory block size: 128MB [ 0.239692] gcov: version magic: 0x41383552 [ 0.241423] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.242138] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.243322] pinctrl core: initialized pinctrl subsystem [ 0.244261] [ 0.244858] ************************************************************* [ 0.245019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.246020] ** ** [ 0.247026] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.248021] ** ** [ 0.249022] ** This means that this kernel is built to expose internal ** [ 0.250022] ** IOMMU data structures, which may compromise security on ** [ 0.251021] ** your system. ** [ 0.252030] ** ** [ 0.253020] ** If you see this message and you are not debugging the ** [ 0.254022] ** kernel, report this immediately to your vendor! ** [ 0.255022] ** ** [ 0.256017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.257023] ************************************************************* [ 0.258896] NET: Registered protocol family 16 [ 0.259532] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.260098] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.261085] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.263031] cpuidle: using governor menu [ 0.265492] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.269831] PCI: Using configuration type 1 for base access [ 0.272450] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.285056] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.286027] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.288180] cryptd: max_cpu_qlen set to 1000 [ 0.290259] ACPI: Added _OSI(Module Device) [ 0.291016] ACPI: Added _OSI(Processor Device) [ 0.292016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.293014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.296911] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.298625] ACPI: Interpreter enabled [ 0.299065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.300013] ACPI: Using IOAPIC for interrupt routing [ 0.301203] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.302329] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.312534] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.313056] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.314029] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.315124] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.317070] acpiphp: Slot [2] registered [ 0.318205] acpiphp: Slot [5] registered [ 0.319150] acpiphp: Slot [6] registered [ 0.320310] acpiphp: Slot [7] registered [ 0.322124] acpiphp: Slot [8] registered [ 0.323083] acpiphp: Slot [9] registered [ 0.324082] acpiphp: Slot [10] registered [ 0.325155] acpiphp: Slot [3] registered [ 0.326072] acpiphp: Slot [4] registered [ 0.327008] acpiphp: Slot [11] registered [ 0.327895] acpiphp: Slot [12] registered [ 0.330110] acpiphp: Slot [13] registered [ 0.331121] acpiphp: Slot [14] registered [ 0.333167] acpiphp: Slot [15] registered [ 0.334153] acpiphp: Slot [16] registered [ 0.336153] acpiphp: Slot [17] registered [ 0.337132] acpiphp: Slot [18] registered [ 0.339150] acpiphp: Slot [19] registered [ 0.340131] acpiphp: Slot [20] registered [ 0.342139] acpiphp: Slot [21] registered [ 0.343165] acpiphp: Slot [22] registered [ 0.345138] acpiphp: Slot [23] registered [ 0.346134] acpiphp: Slot [24] registered [ 0.347181] acpiphp: Slot [25] registered [ 0.349131] acpiphp: Slot [26] registered [ 0.350119] acpiphp: Slot [27] registered [ 0.352170] acpiphp: Slot [28] registered [ 0.353138] acpiphp: Slot [29] registered [ 0.355117] acpiphp: Slot [30] registered [ 0.356113] acpiphp: Slot [31] registered [ 0.357128] PCI host bridge to bus 0000:00 [ 0.359023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.361031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.364029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.366033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.368031] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.371035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.373228] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.377396] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.379407] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.389789] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.394567] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.397023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.399027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.401022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.404693] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.408083] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.411051] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.414734] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.422025] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.436020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.442021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.447834] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.475030] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.500027] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.571029] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.584086] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.593024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.604022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.627023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.638162] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.649020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.659019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.684020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.698245] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.706016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.715019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.729021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.742463] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.754023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.762020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.789111] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.805544] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.815020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.829085] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.854030] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.869102] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.873358] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.875388] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.877529] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.879339] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.884229] iommu: Default domain type: Passthrough [ 0.886428] SCSI subsystem initialized [ 0.888275] ACPI: bus type USB registered [ 0.890150] usbcore: registered new interface driver usbfs [ 0.893161] usbcore: registered new interface driver hub [ 0.895122] usbcore: registered new device driver usb [ 0.897282] pps_core: LinuxPPS API ver. 1 registered [ 0.900017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.904091] PTP clock support registered [ 0.906147] EDAC MC: Ver: 3.0.0 [ 0.908131] PCI: Using ACPI for IRQ routing [ 0.909645] NetLabel: Initializing [ 0.910010] NetLabel: domain hash size = 128 [ 0.910863] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.912098] NetLabel: unlabeled traffic allowed by default [ 0.914128] vgaarb: loaded [ 0.915232] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.916017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.923037] clocksource: Switched to clocksource kvm-clock [ 1.032258] VFS: Disk quotas dquot_6.6.0 [ 1.034087] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.036322] *** VALIDATE ramfs *** [ 1.037400] *** VALIDATE hugetlbfs *** [ 1.038587] pnp: PnP ACPI init [ 1.040619] pnp: PnP ACPI: found 6 devices [ 1.057811] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.061136] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.063313] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.065508] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.067785] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.069887] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.072869] NET: Registered protocol family 2 [ 1.075427] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.080505] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.084723] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.089552] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.093268] TCP: Hash tables configured (established 65536 bind 65536) [ 1.096346] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.099525] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.102652] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.105776] NET: Registered protocol family 1 [ 1.108134] RPC: Registered named UNIX socket transport module. [ 1.110623] RPC: Registered udp transport module. [ 1.112449] RPC: Registered tcp transport module. [ 1.114303] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.116837] NET: Registered protocol family 44 [ 1.118785] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.121461] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.123707] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.126284] PCI: CLS 0 bytes, default 64 [ 1.128202] Unpacking initramfs... [ 2.504714] debug: unmapping init [mem 0xffff94603cc54000-0xffff94603ffbffff] [ 2.508761] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.511162] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.514165] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.989270] Initialise system trusted keyrings [ 2.991174] Key type blacklist registered [ 2.993042] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.002096] zbud: loaded [ 3.005399] *** VALIDATE nfs *** [ 3.006701] *** VALIDATE nfs4 *** [ 3.008294] pstore: using deflate compression [ 3.010555] Platform Keyring initialized [ 3.097937] NET: Registered protocol family 38 [ 3.099166] Key type asymmetric registered [ 3.100284] Asymmetric key parser 'x509' registered [ 3.102326] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.106396] io scheduler mq-deadline registered [ 3.108273] io scheduler kyber registered [ 3.110088] io scheduler bfq registered [ 3.112132] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.115409] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.125599] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.128661] ACPI: Power Button [PWRF] [ 3.139530] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.145627] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.160412] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.168414] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.191755] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.221220] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.254297] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.261555] Non-volatile memory driver v1.3 [ 3.263757] Linux agpgart interface v0.103 [ 3.314461] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.317322] vda: detected capacity change from 0 to 74702848 [ 3.335310] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.338729] vdb: detected capacity change from 0 to 1073741824 [ 3.356045] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.359569] vdc: detected capacity change from 0 to 2621440000 [ 3.381374] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.384642] vdd: detected capacity change from 0 to 2621440000 [ 3.409430] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.411958] vde: detected capacity change from 0 to 4294967296 [ 3.427457] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.430634] vdf: detected capacity change from 0 to 4294967296 [ 3.439688] libphy: Fixed MDIO Bus: probed [ 3.451650] usbcore: registered new interface driver usbserial_generic [ 3.456190] usbserial: USB Serial support registered for generic [ 3.458324] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.461622] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.462824] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.465359] mousedev: PS/2 mouse device common for all mice [ 3.468837] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.470892] rtc_cmos 00:05: RTC can wake from S4 [ 3.474609] rtc_cmos 00:05: registered as rtc0 [ 3.476827] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.479510] intel_pstate: CPU model not supported [ 3.483817] hid: raw HID events driver (C) Jiri Kosina [ 3.484511] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.489735] usbcore: registered new interface driver usbhid [ 3.489742] usbhid: USB HID core driver [ 3.489895] drop_monitor: Initializing network drop monitor service [ 3.490067] Initializing XFRM netlink socket [ 3.490432] NET: Registered protocol family 10 [ 3.491938] Segment Routing with IPv6 [ 3.495922] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.496704] NET: Registered protocol family 17 [ 3.509741] mpls_gso: MPLS GSO support [ 3.514637] RAS: Correctable Errors collector initialized. [ 3.517645] AVX version of gcm_enc/dec engaged. [ 3.519228] AES CTR mode by8 optimization enabled [ 3.610432] sched_clock: Marking stable (3610387809, 0)->(4670654371, -1060266562) [ 3.614429] registered taskstats version 1 [ 3.616603] Loading compiled-in X.509 certificates [ 3.620055] zswap: loaded using pool lzo/zbud [ 3.651078] Key type big_key registered [ 3.672101] Key type encrypted registered [ 3.674212] ima: No TPM chip found, activating TPM-bypass! [ 3.676804] ima: Allocated hash algorithm: sha1 [ 3.678661] ima: No architecture policies found [ 3.680841] evm: Initialising EVM extended attributes: [ 3.682766] evm: security.selinux [ 3.684187] evm: security.ima [ 3.685249] evm: security.capability [ 3.686584] evm: HMAC attrs: 0x1 [ 3.691252] rtc_cmos 00:05: setting system clock to 2026-08-17 17:43:02 UTC (1786988582) [ 3.698877] debug: unmapping init [mem 0xffffffff91e03000-0xffffffff91ffffff] [ 3.702229] debug: unmapping init [mem 0xffffffff90b82000-0xffffffff90e58fff] [ 3.711122] Write protecting the kernel read-only data: 28672k [ 3.714859] debug: unmapping init [mem 0xffffffff8f203000-0xffffffff8f3fffff] [ 3.717786] debug: unmapping init [mem 0xffffffff8fb14000-0xffffffff8fbfffff] [ 3.761482] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.771950] systemd[1]: Detected virtualization kvm. [ 3.774335] systemd[1]: Detected architecture x86-64. [ 3.775957] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.806514] systemd[1]: No hostname configured. [ 3.808071] systemd[1]: Set hostname to . [ 3.810379] random: systemd: uninitialized urandom read (16 bytes read) [ 3.813052] systemd[1]: Initializing machine ID from random generator. [ 3.851903] random: ln: uninitialized urandom read (6 bytes read) [ 3.978535] random: systemd: uninitialized urandom read (16 bytes read) [ 3.981449] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.986308] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.989914] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.710282] device-mapper: uevent: version 1.0.3 [ 4.712673] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 5.532787] random: fast init done [ 5.542330] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.605663] scsi host0: ata_piix [ 5.665400] scsi host1: ata_piix [ 5.666961] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.668930] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.366361] dracut-initqueue[592]: RTNETLINK answers: File exists [ 10.194960] random: crng init done [ 10.195810] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.785567] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.057949] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.337850] SELinux: Disabled at runtime. [ 12.402806] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.410406] systemd[1]: Detected virtualization kvm. [ 12.412662] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.968513] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.973067] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.978524] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.985531] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.989368] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.997267] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.006668] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... [ 13.038980] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice User and Session Slice. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.509529] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.046961] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.061146] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.212911] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.233716] EDAC sbridge: Ver: 1.1.2 [ 16.170830] Key type dns_resolver registered [ 16.481096] NFS: Registering the id_resolver key type [ 16.483135] Key type id_resolver registered [ 16.484374] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg241-server login: [ 59.614646] libcfs: loading out-of-tree module taints kernel. [ 59.657332] Key type ._llcrypt registered [ 59.660209] Key type .llcrypt registered [ 59.847432] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 78.573912] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 80.351572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 80.367071] alg: No test for adler32 (adler32-zlib) [ 82.052818] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 83.179021] hrtimer: interrupt took 3747243 ns [ 83.382873] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 85.226375] Key type lgssc registered [ 86.704248] Lustre: Echo OBD driver; http://www.lustre.org/ [ 102.232264] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 154.695609] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 170.105255] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 170.161660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 171.398616] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 171.413765] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 171.527209] Lustre: lustre-MDT0000: new disk, initializing [ 171.636139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 171.653045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 176.924189] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 191.647335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 191.775925] Lustre: 6515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 191.822488] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 191.826428] Lustre: Skipped 1 previous similar message [ 191.954808] Lustre: lustre-MDT0001: new disk, initializing [ 192.063524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 192.106181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 192.142549] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 197.593313] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.030989] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 214.881744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 215.192159] Lustre: lustre-OST0000: new disk, initializing [ 215.206407] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 215.234499] Lustre: 8453:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 215.308500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 220.753633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 220.763516] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 220.805903] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 222.009137] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.760872] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 239.962486] Lustre: lustre-OST0001: new disk, initializing [ 239.971285] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 239.980359] Lustre: 9525:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 240.107150] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 247.624909] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 250.004753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 250.009956] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 250.124679] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 261.809744] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 270.432228] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 278.426633] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing check_logdir /tmp/testlogs/ [ 284.844565] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing yml_node [ 290.396475] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 292.860034] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 296.201636] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 297.988326] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 17 13:47:54 EDT 2026 [ 319.409972] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 321.575339] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 337.381490] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 337.385607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 337.400427] Lustre: Skipped 1 previous similar message [ 337.416083] Lustre: Skipped 3 previous similar messages [ 342.498190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 342.503221] Lustre: Skipped 3 previous similar messages [ 343.732226] Lustre: server umount lustre-MDT0000 complete [ 347.167666] LustreError: 7462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786988925 with bad export cookie 5991923957547362309 [ 347.176186] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 347.177914] LustreError: 7462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 347.622677] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 347.630137] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 347.648331] Lustre: Skipped 3 previous similar messages [ 347.656682] Lustre: Skipped 1 previous similar message [ 352.736859] LustreError: 11156:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 352.739018] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 352.763457] LustreError: 11156:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 352.797484] Lustre: Skipped 1 previous similar message [ 353.733309] Lustre: server umount lustre-MDT0001 complete [ 358.466244] Lustre: server umount lustre-OST0000 complete [ 362.346940] Lustre: server umount lustre-OST0001 complete [ 369.806355] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 379.154906] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 417.032946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 417.245205] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 417.276710] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 417.341729] Lustre: lustre-MDT0000: new disk, initializing [ 417.414605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 417.436438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 422.288776] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 433.905686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 434.146976] Lustre: 15082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 434.269464] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 434.273735] Lustre: Skipped 1 previous similar message [ 434.372884] Lustre: lustre-MDT0001: new disk, initializing [ 434.540881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 434.592035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 434.606708] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 440.265701] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 451.224643] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 457.828382] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 465.181469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 465.495120] Lustre: lustre-OST0000: new disk, initializing [ 465.501084] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 465.509856] Lustre: 17050:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 465.663477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 466.913201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 466.927208] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 467.173469] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 472.857358] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 485.066766] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 490.794808] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 491.162882] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 496.076182] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 496.313284] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 502.753547] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 502.770660] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 502.807091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 503.265516] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 505.026472] Lustre: server umount lustre-OST0000 complete [ 511.458927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 511.467756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 511.480368] Lustre: Skipped 1 previous similar message [ 511.485731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 511.497494] Lustre: Skipped 2 previous similar messages [ 516.245902] Lustre: server umount lustre-MDT0000 complete [ 518.626066] LustreError: 15094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 518.645646] LustreError: 15094:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 520.557635] LustreError: 15073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786989099 with bad export cookie 5991923957547363849 [ 520.565752] LustreError: 15073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 520.566243] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 520.982643] Lustre: server umount lustre-MDT0001 complete [ 527.747020] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 13:51:43 (1786989103) [ 551.544653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 552.216391] LustreError: 19927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 552.293380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 557.539476] LustreError: 19928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.104432] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 562.658858] LustreError: 19927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 568.916783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 575.434977] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 584.619121] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 591.913980] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 594.208518] Lustre: Failing over lustre-MDT0000 [ 594.595396] Lustre: server umount lustre-MDT0000 complete [ 594.923883] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 594.928765] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 594.945937] Lustre: Skipped 1 previous similar message [ 594.959420] LustreError: 19927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 594.974063] LustreError: 19927:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 613.350875] LustreError: 19928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 613.369750] LustreError: 19928:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 615.137111] Lustre: 3649:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786989177/real 1786989177] req@ffff9460afef9f80 x1873793423414528/t0(0) o400->MGC192.168.202.141@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786989193 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 615.162837] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 615.733268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 624.610753] LustreError: 3645:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9460aff4df80 x1873793423416960/t0(0) o250->MGC192.168.202.141@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 624.979365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 624.988797] Lustre: Skipped 1 previous similar message [ 625.082040] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 629.951646] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 630.242748] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 630.255095] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 630.287753] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 641.138092] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 645.601720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 645.612773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 645.621330] Lustre: Skipped 1 previous similar message [ 645.625631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 645.628675] Lustre: Skipped 2 previous similar messages [ 652.187797] Lustre: server umount lustre-MDT0000 complete [ 653.793019] LustreError: 19927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 653.812687] LustreError: 19927:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 656.751087] LustreError: 19909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786989235 with bad export cookie 5991923957547365263 [ 656.774970] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 657.142796] Lustre: server umount lustre-MDT0001 complete [ 667.819038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 674.252271] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 684.923770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 689.009344] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 698.365492] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 705.062442] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 707.475901] Lustre: Failing over lustre-MDT0000 [ 707.777803] Lustre: server umount lustre-MDT0000 complete [ 710.570335] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 710.587600] Lustre: Skipped 2 previous similar messages [ 720.866398] LustreError: 23267:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 720.895318] LustreError: 23267:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 727.024350] Lustre: 3647:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786989289/real 1786989289] req@ffff9460aff4dc00 x1873793423444864/t0(0) o400->MGC192.168.202.141@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786989305 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 727.060966] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 728.602160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 736.641852] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 741.875260] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 741.885140] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 741.896264] Lustre: Skipped 1 previous similar message [ 741.932484] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 742.426521] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 754.292006] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 760.292143] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 760.308365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 760.317926] Lustre: Skipped 3 previous similar messages [ 765.397668] Lustre: server umount lustre-MDT0000 complete [ 770.055066] LustreError: 23246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786989348 with bad export cookie 5991923957547366201 [ 770.057505] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 770.075569] LustreError: 23246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 770.541259] Lustre: server umount lustre-MDT0001 complete [ 780.975861] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 13:55:57 (1786989357) [ 801.122458] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 813.092415] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 857.766931] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 870.208511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 870.427869] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 870.450477] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 870.596461] Lustre: lustre-MDT0000: new disk, initializing [ 870.725292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 870.728160] Lustre: Skipped 3 previous similar messages [ 870.759979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 876.025367] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 888.130270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 888.365689] Lustre: 29643:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 888.813582] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 893.735834] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 904.989547] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 912.138331] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 921.689111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 922.004578] Lustre: 31616:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 923.521763] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 923.568055] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 923.640174] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 923.663531] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 929.361622] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 939.873933] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 946.606733] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 969.697982] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 969.703151] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 969.713839] LustreError: Skipped 1 previous similar message [ 969.714848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 969.758189] Lustre: Skipped 2 previous similar messages [ 974.846780] Lustre: server umount lustre-OST0000 complete [ 993.248157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 993.609550] Lustre: server umount lustre-MDT0000 complete [ 995.301708] LustreError: 29651:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 995.338323] LustreError: 29651:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 998.173022] LustreError: 31615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786989576 with bad export cookie 5991923957547366607 [ 998.197128] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 998.202700] LustreError: 31615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 998.851727] Lustre: server umount lustre-MDT0001 complete [ 1013.884926] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 1016.786385] Key type lgssc unregistered [ 1017.041940] LNet: 33600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1017.049229] LNetError: 33600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1017.068205] LNet: Removed LNI 192.168.202.141@tcp [ 1017.851165] Key type .llcrypt unregistered [ 1017.853887] Key type ._llcrypt unregistered [ 1043.531107] Key type ._llcrypt registered [ 1043.535107] Key type .llcrypt registered [ 1043.641048] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 1044.621448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1044.685232] alg: No test for adler32 (adler32-zlib) [ 1045.771515] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 1045.974603] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 1047.616628] Key type lgssc registered [ 1048.723224] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1057.503830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1057.531482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1059.058729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1064.140086] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1068.517425] LustreError: 35017:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1068.531281] LustreError: 35017:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1073.550951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1073.635935] LustreError: 35018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1074.048385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1078.733325] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1087.973105] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1097.300873] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1109.237438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1109.701531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1118.310680] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1118.927289] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1127.193517] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1129.190706] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1148.314241] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 14:02:04 (1786989724) [ 1151.080651] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1153.397697] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1163.621983] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 14:02:19 (1786989739) [ 1177.253415] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 14:02:33 (1786989753) [ 1178.965454] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1189.505694] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1189.513626] Lustre: Skipped 2 previous similar messages [ 1191.589850] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 14:02:48 (1786989768) [ 1193.620313] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1193.622879] Lustre: Skipped 2 previous similar messages [ 1203.494533] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1203.503144] Lustre: Skipped 4 previous similar messages [ 1220.742954] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 14:03:17 (1786989797) [ 1223.838388] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1228.303590] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1239.154418] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 14:03:35 (1786989815) [ 1241.990389] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1241.997798] Lustre: Skipped 6 previous similar messages [ 1257.988407] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 14:03:54 (1786989834) [ 1277.930249] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1277.938984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1277.939244] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1277.955480] Lustre: Skipped 1 previous similar message [ 1282.599847] Lustre: server umount lustre-OST0000 complete [ 1288.163815] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1288.186423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1288.193870] Lustre: Skipped 2 previous similar messages [ 1293.267254] Lustre: server umount lustre-MDT0000 complete [ 1293.294495] LustreError: 35017:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1293.337490] LustreError: 35017:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1297.389273] LustreError: 35013:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1297.416714] LustreError: 35013:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1297.994704] LustreError: 34997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786989876 with bad export cookie 4834766863002121312 [ 1297.999101] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1298.005477] LustreError: 34997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1298.353086] Lustre: server umount lustre-MDT0001 complete [ 1312.621187] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 1315.507987] Key type lgssc unregistered [ 1315.939853] LNet: 42895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1315.945058] LNetError: 42895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1315.959387] LNet: Removed LNI 192.168.202.141@tcp [ 1316.821601] Key type .llcrypt unregistered [ 1316.825535] Key type ._llcrypt unregistered [ 1341.014926] Key type ._llcrypt registered [ 1341.017569] Key type .llcrypt registered [ 1341.143357] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 1358.445175] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 1359.177608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1359.198207] alg: No test for adler32 (adler32-zlib) [ 1360.169423] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 1360.355429] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 1362.040355] Key type lgssc registered [ 1362.862251] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1405.364305] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1405.413284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1406.679420] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1406.721059] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1406.853473] Lustre: lustre-MDT0000: new disk, initializing [ 1406.996610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1407.028636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1412.234053] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1426.845238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1427.025643] Lustre: 46061:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1427.108086] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1427.114902] Lustre: Skipped 1 previous similar message [ 1427.273395] Lustre: lustre-MDT0001: new disk, initializing [ 1427.437357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1427.515514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1427.534452] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1433.990576] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1446.594039] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1453.926667] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1462.341629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1462.709717] Lustre: lustre-OST0000: new disk, initializing [ 1462.712445] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1462.720287] Lustre: 48031:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1462.822960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1468.572129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1468.598305] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1468.802260] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1471.270852] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1482.512908] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1488.270937] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1488.508941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1494.039177] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1494.214280] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1499.619388] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1499.629237] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1499.662636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1504.766442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1504.783896] Lustre: Skipped 2 previous similar messages [ 1509.861904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1509.881555] Lustre: Skipped 1 previous similar message [ 1511.392105] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1511.676105] Lustre: server umount lustre-OST0000 complete [ 1519.589144] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1519.596459] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1519.613754] Lustre: Skipped 1 previous similar message [ 1519.620351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1522.947948] Lustre: server umount lustre-MDT0000 complete [ 1527.103971] LustreError: 46855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786990105 with bad export cookie 5578385981996240781 [ 1527.111066] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1527.112874] LustreError: 46855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1527.663791] Lustre: server umount lustre-MDT0001 complete [ 1539.213561] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 14:08:35 (1786990115) [ 1547.140794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1547.710918] LustreError: 50491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1547.732713] LustreError: 50491:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1547.808929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1552.880653] LustreError: 50492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1553.536856] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1557.986365] LustreError: 50491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1563.110448] LustreError: 50492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1563.657677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1564.156418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1569.413427] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1577.806406] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1584.101635] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1590.778322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1591.165290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1598.529400] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1607.295821] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1612.714344] LustreError: 53020:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1613.950159] Lustre: Modifying parameter general.timeout=1 in log params [ 1614.987289] Lustre: Modifying parameter general.timeout=2 in log params [ 1617.218989] Lustre: Modifying parameter general.timeout=4 in log params [ 1617.226366] Lustre: Skipped 1 previous similar message [ 1621.585061] Lustre: Modifying parameter general.timeout=8 in log params [ 1621.588021] Lustre: Skipped 3 previous similar messages [ 1630.879716] Lustre: Modifying parameter general.timeout=16 in log params [ 1630.884220] Lustre: Skipped 7 previous similar messages [ 1647.137794] Lustre: Modifying parameter general.timeout=31 in log params [ 1647.141115] Lustre: Skipped 14 previous similar messages [ 1679.662184] Lustre: Modifying parameter general.timeout=68 in log params [ 1679.667311] Lustre: Skipped 36 previous similar messages [ 1750.255488] Lustre: Disabling parameter general.timeout= in log params [ 1750.259258] Lustre: Skipped 61 previous similar messages [ 1754.004654] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 14:12:10 (1786990330) [ 1756.908724] Lustre: 59980:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1758.966237] Lustre: 60078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1761.031741] Lustre: 60177:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1766.187482] Lustre: 60393:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1766.196553] Lustre: 60393:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1800.407941] Lustre: 61852:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1800.414231] Lustre: 61852:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1800.448341] LustreError: 61852:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1805.213268] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 14:13:01 (1786990381) [ 1823.205215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1823.209841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1823.217205] Lustre: Skipped 3 previous similar messages [ 1823.223019] Lustre: Skipped 3 previous similar messages [ 1825.249826] Lustre: server umount lustre-MDT0000 complete [ 1828.324824] LustreError: 51781:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1828.362029] LustreError: 51781:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1830.634759] LustreError: 50471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786990409 with bad export cookie 5578385981996241663 [ 1830.648415] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1830.658416] LustreError: 50471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1831.841644] Lustre: server umount lustre-MDT0001 complete [ 1849.040040] Lustre: server umount lustre-OST0000 complete [ 1878.990638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1879.355681] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1879.392665] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1879.407701] Lustre: Skipped 22 previous similar messages [ 1879.679424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1884.908918] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1894.815380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1894.901858] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1894.939869] Lustre: 64385:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1900.589456] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1908.508456] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1914.613594] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1923.091899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1923.391572] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1924.028414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1924.044325] Lustre: Skipped 1 previous similar message [ 1930.879023] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1940.632797] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1942.795212] Lustre: 66953:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1953.251158] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1953.252801] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1953.254855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1953.254867] Lustre: Skipped 1 previous similar message [ 1953.273599] LustreError: Skipped 1 previous similar message [ 1956.625992] Lustre: server umount lustre-OST0000 complete [ 1961.954402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1961.961077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1961.969191] Lustre: Skipped 1 previous similar message [ 1973.742363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1973.761401] Lustre: Skipped 7 previous similar messages [ 1975.264689] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1975.598955] Lustre: server umount lustre-MDT0000 complete [ 1978.850549] LustreError: 64397:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1978.874929] LustreError: 64397:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1979.759041] LustreError: 64378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786990558 with bad export cookie 5578385981996244785 [ 1979.771160] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1979.772509] LustreError: 64378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1980.394475] Lustre: server umount lustre-MDT0001 complete [ 1993.004625] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 1996.751830] Key type lgssc unregistered [ 1997.155861] LNet: 68168:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1997.165688] LNetError: 68168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1997.179277] LNet: Removed LNI 192.168.202.141@tcp [ 1998.246868] Key type .llcrypt unregistered [ 1998.249191] Key type ._llcrypt unregistered [ 2026.216726] Key type ._llcrypt registered [ 2026.219682] Key type .llcrypt registered [ 2026.374742] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 2027.973521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2028.037239] alg: No test for adler32 (adler32-zlib) [ 2029.254349] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 2029.758266] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 2031.528333] Key type lgssc registered [ 2032.827397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2042.030215] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2042.070871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2043.869952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2049.618758] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2054.116712] LustreError: 69581:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2054.144055] LustreError: 69581:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2055.145717] LustreError: 69582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2059.833274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2060.739321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2065.172810] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2073.837473] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2079.252092] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2087.089197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2087.905122] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2095.466797] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2104.513125] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2113.663988] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2125.796645] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2125.802381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2125.825545] Lustre: Skipped 1 previous similar message [ 2129.702710] Lustre: server umount lustre-MDT0000 complete [ 2130.914375] LustreError: 69581:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2130.929123] LustreError: 69581:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2134.036636] LustreError: 69563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786990712 with bad export cookie 17287570516929879965 [ 2134.045980] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2134.054863] LustreError: 69563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2134.653451] Lustre: server umount lustre-MDT0001 complete [ 2148.711302] Lustre: server umount lustre-OST0000 complete [ 2177.569445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2177.929581] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2177.976497] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2178.386117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2183.686695] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2192.937213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2193.142469] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2193.202149] Lustre: 74454:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2193.948703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2198.993445] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2206.667662] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2213.347317] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2220.930084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2221.407670] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2221.908769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2229.414587] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2237.715795] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2239.622809] Lustre: 77030:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2241.637735] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2249.704167] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2249.710414] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2249.712326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2249.712333] Lustre: Skipped 2 previous similar messages [ 2249.717153] LustreError: Skipped 1 previous similar message [ 2249.739564] Lustre: Skipped 2 previous similar messages [ 2253.208874] Lustre: server umount lustre-OST0000 complete [ 2259.939872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2259.949627] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2259.963197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2259.990919] Lustre: Skipped 1 previous similar message [ 2260.000641] Lustre: Skipped 1 previous similar message [ 2263.391214] Lustre: server umount lustre-MDT0000 complete [ 2265.061220] LustreError: 74465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2265.073247] LustreError: 74465:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2267.142681] LustreError: 74446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786990845 with bad export cookie 17287570516929880812 [ 2267.160375] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2267.497853] Lustre: server umount lustre-MDT0001 complete [ 2281.131530] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 2284.746867] Key type lgssc unregistered [ 2285.199870] LNet: 78243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2285.222662] LNetError: 78243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2285.248988] LNet: Removed LNI 192.168.202.141@tcp [ 2286.385406] Key type .llcrypt unregistered [ 2286.387315] Key type ._llcrypt unregistered [ 2317.031915] Key type ._llcrypt registered [ 2317.034766] Key type .llcrypt registered [ 2317.214259] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 2318.687437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2318.848839] alg: No test for adler32 (adler32-zlib) [ 2320.145245] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 2320.456340] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 2322.241326] Key type lgssc registered [ 2323.299890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2331.812193] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2331.845721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2333.490740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2338.815569] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2343.909316] LustreError: 79659:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2343.940223] LustreError: 79659:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2348.658602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2349.392546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2354.202227] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2363.215671] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2370.154454] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2377.073124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2377.753190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2384.140671] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2391.943189] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2398.587792] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2399.605887] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2401.567020] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2401.570567] Lustre: Skipped 1 previous similar message [ 2404.233660] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2404.238739] Lustre: Skipped 2 previous similar messages [ 2408.728730] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2408.737791] Lustre: Skipped 4 previous similar messages [ 2417.687784] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2417.697963] Lustre: Skipped 9 previous similar messages [ 2434.339398] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2434.346291] Lustre: Skipped 17 previous similar messages [ 2466.924396] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2466.930539] Lustre: Skipped 36 previous similar messages [ 2497.589893] Lustre: 87760:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2500.432975] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 14:24:37 (1786991077) [ 2502.026182] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2503.958700] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 14:24:40 (1786991080) [ 2511.843025] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2511.852621] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2511.873206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2511.887581] Lustre: Skipped 1 previous similar message [ 2514.262475] Lustre: server umount lustre-OST0000 complete [ 2517.986460] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2517.993590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2518.006946] Lustre: Skipped 1 previous similar message [ 2518.011567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2522.081962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2522.086046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2524.066879] Lustre: server umount lustre-MDT0000 complete [ 2527.213389] LustreError: 79660:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2527.245446] LustreError: 79660:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2527.807299] LustreError: 82249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786991106 with bad export cookie 747837600855811849 [ 2527.811820] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2527.813958] LustreError: 82249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2528.241726] Lustre: server umount lustre-MDT0001 complete [ 2539.862818] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 2542.772821] Key type lgssc unregistered [ 2543.063394] LNet: 89013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2543.069828] LNetError: 89013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2544.113511] LNet: Removed LNI 192.168.202.141@tcp [ 2545.079482] Key type .llcrypt unregistered [ 2545.081743] Key type ._llcrypt unregistered [ 2550.734793] Key type ._llcrypt registered [ 2550.738506] Key type .llcrypt registered [ 2550.901753] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2556.373896] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules [ 2557.925348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2557.994279] alg: No test for adler32 (adler32-zlib) [ 2559.053558] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 2559.361692] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 2561.024201] Key type lgssc registered [ 2562.167651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2563.167780] LustreError: 90215:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2577.944087] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 2585.408460] LustreError: 90215:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2586.845609] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2586.882687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2588.520389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2593.613088] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2598.889270] LustreError: 91239:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2598.915557] LustreError: 91239:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2604.013320] LustreError: 91239:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2604.602818] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 14:26:20 (1786991180) [ 2609.128963] LustreError: 91238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2611.822119] LustreError: 91270:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2611.833398] LustreError: 91270:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2611.843820] LustreError: 91270:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 23, retries 0, failed: rc = -5 [ 2612.326189] Lustre: server umount lustre-MDT0000 complete [ 2628.161436] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 2630.905406] Key type lgssc unregistered [ 2631.273939] LNet: 92768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2631.277182] LNetError: 92768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2631.298558] LNet: Removed LNI 192.168.202.141@tcp [ 2632.053311] Key type .llcrypt unregistered [ 2632.056840] Key type ._llcrypt unregistered [ 2649.993670] Key type ._llcrypt registered [ 2649.998339] Key type .llcrypt registered [ 2650.163742] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 2651.502229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2651.511934] alg: No test for adler32 (adler32-zlib) [ 2652.654845] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 2652.959181] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 2654.672631] Key type lgssc registered [ 2656.146782] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2664.760292] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2664.786267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2666.386479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2671.413705] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2676.721728] LustreError: 93999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2676.749156] LustreError: 93999:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2680.513702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2681.085682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2685.421764] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2693.530027] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2699.240197] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2707.179359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2707.919888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2715.061362] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2723.180871] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2773.187124] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 14:29:09 (1786991349) [ 2775.509742] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2778.104508] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 14:29:13 (1786991353) [ 2786.803387] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2786.810884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2786.830821] Lustre: Skipped 1 previous similar message [ 2786.861102] Lustre: Skipped 2 previous similar messages [ 2790.723550] Lustre: server umount lustre-MDT0000 complete [ 2791.924048] LustreError: 96450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2791.957577] LustreError: 96450:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2795.995700] LustreError: 93978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786991374 with bad export cookie 10184068966157812445 [ 2796.008698] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2796.026595] LustreError: 93978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2796.431892] Lustre: server umount lustre-MDT0001 complete [ 2811.237254] Lustre: server umount lustre-OST0000 complete [ 2824.338317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2824.684472] LustreError: 98278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2824.843906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2830.305736] LustreError: 98279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2830.438849] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2834.415097] LustreError: 98278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2839.760483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2840.567380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2846.315481] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2854.754969] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2861.680630] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2877.918788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2878.205374] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2878.209801] LustreError: 98266:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2878.224777] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2878.229358] LustreError: 100361:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2878.233533] LustreError: 100361:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2878.237174] LustreError: 100361:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2878.286606] Lustre: server umount lustre-OST0000 complete [ 2878.289391] LustreError: 100361:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2888.722968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2889.009453] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2889.033328] LustreError: 98266:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2889.054409] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2889.062807] LustreError: 100828:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2889.068298] LustreError: 100828:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2889.072835] LustreError: 100828:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2889.145875] Lustre: server umount lustre-OST0000 complete [ 2889.149091] LustreError: 100828:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2900.241951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2900.441858] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2900.447141] Lustre: Found index 0 for lustre-OST0000, updating log [ 2900.457396] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2900.544195] Lustre: lustre-OST0000: new disk, initializing [ 2900.548672] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2900.564189] Lustre: 101362:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2900.881648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2908.099860] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2912.197156] Lustre: Failing over lustre-OST0000 [ 2912.348781] Lustre: server umount lustre-OST0000 complete [ 2916.843385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2916.852982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2916.871161] Lustre: Skipped 1 previous similar message [ 2916.876672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2921.443086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2921.448132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2922.372206] Lustre: server umount lustre-MDT0000 complete [ 2926.568345] LustreError: 98273:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2926.605314] LustreError: 98273:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2926.714483] LustreError: 98258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786991505 with bad export cookie 10184068966157815189 [ 2926.731630] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2936.495233] Lustre: server umount lustre-MDT0001 complete [ 2946.346861] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 14:32:02 (1786991522) [ 2963.954721] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 2974.580679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2974.991728] LustreError: 103715:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2975.081569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2979.999808] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2990.467353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2997.344542] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3000.148217] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3007.571799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3008.186601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3008.200144] Lustre: Skipped 1 previous similar message [ 3008.211968] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3015.340773] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3015.384074] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3015.385728] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3015.536363] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3025.163867] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3025.275864] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3025.424727] Lustre: lustre-OST0001: new disk, initializing [ 3025.434682] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3025.450898] Lustre: 106035:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3032.565619] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3034.256569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3034.274094] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3034.352742] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3041.326475] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3048.453983] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3053.026317] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3053.051188] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3053.080784] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3053.101546] Lustre: Skipped 1 previous similar message [ 3059.028288] Lustre: server umount lustre-MDT0001 complete [ 3060.200302] LustreError: 103711:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3060.229295] LustreError: 103711:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 3069.744631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3069.841907] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3069.843410] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3069.868185] Lustre: 103702:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3071.167057] LustreError: 107604:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3071.173338] LustreError: 107604:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3071.186190] LustreError: 103706:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3071.382654] LustreError: 107618:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.141@tcp: cfg command failed: rc = -17 [ 3071.388181] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3071.388181] [ 3075.574723] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3076.154728] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3090.631564] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 14:34:25 (1786991665) [ 3114.849179] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 3132.105664] Lustre: 109651:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3160.728710] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3165.945337] Lustre: 110787:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3264.482872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3264.494353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3264.513098] Lustre: Skipped 2 previous similar messages [ 3264.521486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3264.525968] Lustre: Skipped 4 previous similar messages [ 3268.111702] Lustre: server umount lustre-MDT0000 complete [ 3270.121307] LustreError: 103710:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3270.152288] LustreError: 103710:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 3272.596888] LustreError: 103697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786991851 with bad export cookie 10184068966157816190 [ 3272.597632] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3272.609343] LustreError: 103697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3273.078541] Lustre: server umount lustre-MDT0001 complete [ 3288.090802] Lustre: server umount lustre-OST0000 complete [ 3291.488090] Lustre: 93311:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786991854/real 1786991854] req@ffff9460bfb18000 x1873796117407744/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786991870 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3291.529044] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3291.542802] Lustre: Skipped 3 previous similar messages [ 3292.329402] Lustre: server umount lustre-OST0001 complete [ 3299.994993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3311.483486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3320.821430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3332.407775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3356.352741] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 3369.651445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3369.673570] Lustre: lustre-MDT0000: reset Object Index mappings [ 3370.284407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3370.287990] Lustre: Skipped 2 previous similar messages [ 3375.166966] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3385.042024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3385.060821] Lustre: lustre-MDT0001: reset Object Index mappings [ 3391.138697] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3394.360516] Lustre: 115892:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3401.723726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3410.363181] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3410.734522] LustreError: 116246:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3410.758940] LustreError: 116246:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3410.792182] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 3414.830476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3420.227026] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3426.314847] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3426.322913] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3428.063887] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3437.388445] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3441.352297] Lustre: 117771:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3488.946297] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 14:41:05 (1786992065) [ 3518.449150] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3518.452237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3518.460322] Lustre: Skipped 2 previous similar messages [ 3518.476860] Lustre: Skipped 6 previous similar messages [ 3521.792993] Lustre: server umount lustre-MDT0000 complete [ 3526.339150] LustreError: 114724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786992105 with bad export cookie 10184068966157863041 [ 3526.359565] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3532.819642] Lustre: server umount lustre-MDT0001 complete [ 3542.558421] Lustre: server umount lustre-OST0000 complete [ 3553.159859] Lustre: server umount lustre-OST0001 complete [ 3561.441181] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 3572.827119] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 3613.722030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3614.111622] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3614.198382] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3614.325770] Lustre: lustre-MDT0000: new disk, initializing [ 3614.432887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3614.441426] Lustre: Skipped 3 previous similar messages [ 3614.465180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3620.357160] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3629.422407] Lustre: server umount lustre-MDT0000 complete [ 3637.984286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3638.362859] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3638.397070] Lustre: 122039:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3638.412247] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3643.449378] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3649.481280] Lustre: server umount lustre-MDT0000 complete [ 3661.668804] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 14:43:57 (1786992237) [ 3663.534514] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3666.194661] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 14:44:02 (1786992242) [ 3688.809070] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 3700.969305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3706.803092] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3716.924303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3717.046996] Lustre: 124092:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3717.131501] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3717.135659] Lustre: Skipped 1 previous similar message [ 3717.208455] Lustre: lustre-MDT0001: new disk, initializing [ 3717.398297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3717.416255] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3722.709212] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3728.925610] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3737.677539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3738.186136] Lustre: lustre-OST0000: new disk, initializing [ 3738.193894] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3738.210378] Lustre: 125669:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3740.062715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3740.076171] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3740.222773] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3746.459830] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3760.030083] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3760.220849] Lustre: lustre-OST0001: new disk, initializing [ 3760.223860] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3760.228630] Lustre: 126541:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3761.500114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3761.517778] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3761.718359] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3768.351823] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3780.120383] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3785.082298] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4214.926093] Lustre: 132967:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4214.939416] Lustre: 132967:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4214.945786] Lustre: 132967:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4214.951135] Lustre: 132967:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4214.955493] Lustre: 132967:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4214.959611] Lustre: 132967:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4534.201197] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 14:58:29 (1786993109) [ 4552.674212] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4552.689393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4552.708111] Lustre: Skipped 3 previous similar messages [ 4552.728672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4552.741258] Lustre: Skipped 4 previous similar messages [ 4558.315118] LustreError: 127347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4558.332042] LustreError: 127347:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 4558.663535] Lustre: server umount lustre-MDT0000 complete [ 4562.675562] LustreError: 124083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786993141 with bad export cookie 10184068966157929905 [ 4562.682078] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4562.687638] LustreError: 124083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4563.431980] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4563.458332] Lustre: Skipped 3 previous similar messages [ 4563.466175] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4563.472910] Lustre: Skipped 5 previous similar messages [ 4569.484058] Lustre: server umount lustre-MDT0001 complete [ 4580.657365] Lustre: server umount lustre-OST0000 complete [ 4599.264144] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4599.579775] Lustre: server umount lustre-OST0001 complete [ 4608.107788] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 4623.016632] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 4670.160805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4670.502898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4670.579586] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4670.787374] Lustre: lustre-MDT0000: new disk, initializing [ 4670.880625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4670.890472] Lustre: Skipped 5 previous similar messages [ 4670.905958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4675.852225] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4688.810813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4688.989250] Lustre: 136043:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4689.039892] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4689.042733] Lustre: Skipped 1 previous similar message [ 4689.124364] Lustre: lustre-MDT0001: new disk, initializing [ 4689.394887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4689.415062] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4694.524767] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4706.509155] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4713.949967] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4724.494123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4725.126087] Lustre: lustre-OST0000: new disk, initializing [ 4725.131252] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4725.137383] Lustre: 138023:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4725.296244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4725.301348] Lustre: Skipped 1 previous similar message [ 4726.813877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4726.822762] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4727.006733] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4733.017279] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4742.655594] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4754.193925] Lustre: lustre-MDD0000: changelog on [ 4756.999871] Lustre: lustre-MDD0001: changelog on [ 6827.742054] Lustre: lustre-MDD0001: changelog off [ 6830.269513] Lustre: 136048:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6831.452332] Lustre: lustre-MDD0000: changelog off [ 6839.780338] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6839.785765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6839.789961] Lustre: Skipped 2 previous similar messages [ 6839.815726] Lustre: Skipped 3 previous similar messages [ 6844.907220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6844.918273] Lustre: Skipped 1 previous similar message [ 6845.970936] Lustre: server umount lustre-OST0000 complete [ 6855.138050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6855.144772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6855.162582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6855.167282] Lustre: Skipped 1 previous similar message [ 6857.712942] Lustre: server umount lustre-MDT0000 complete [ 6860.257696] LustreError: 136048:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6860.274113] LustreError: 136048:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6862.433771] LustreError: 136033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786995441 with bad export cookie 10184068966158757564 [ 6862.441710] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6862.453386] LustreError: 136033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6863.050207] Lustre: server umount lustre-MDT0001 complete [ 6877.273591] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 6880.302872] Key type lgssc unregistered [ 6880.651561] LNet: 147181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6880.659832] LNetError: 147181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6880.673082] LNet: Removed LNI 192.168.202.141@tcp [ 6881.717276] Key type .llcrypt unregistered [ 6881.718435] Key type ._llcrypt unregistered [ 6905.943712] Key type ._llcrypt registered [ 6905.950245] Key type .llcrypt registered [ 6906.131323] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 6928.153805] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 6930.252813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6930.322322] alg: No test for adler32 (adler32-zlib) [ 6931.382353] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 6931.725370] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 6933.552184] Key type lgssc registered [ 6935.016722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6983.807582] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6983.847057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6985.288720] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6985.325671] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6985.409898] Lustre: lustre-MDT0000: new disk, initializing [ 6985.516755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6985.547345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6991.156959] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7004.227348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7004.329489] Lustre: 150389:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7004.379781] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7004.383113] Lustre: Skipped 1 previous similar message [ 7004.464973] Lustre: lustre-MDT0001: new disk, initializing [ 7004.647387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7004.724064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7004.743035] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7010.515741] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7022.561030] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7030.944421] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7039.850535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7040.296573] Lustre: lustre-OST0000: new disk, initializing [ 7040.299685] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7040.304612] Lustre: 152360:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7040.394782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7047.577628] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7047.773987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7047.798692] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7048.040697] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7059.961886] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7066.889309] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_module obdecho/obdecho [ 7069.358926] LustreError: 153364:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7069.370555] LustreError: 153364:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7071.568073] LustreError: 153460:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7089.121918] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7089.126886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7089.136904] Lustre: Skipped 1 previous similar message [ 7089.160647] Lustre: Skipped 2 previous similar messages [ 7093.994711] Lustre: server umount lustre-MDT0000 complete [ 7098.720958] LustreError: 152359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786995677 with bad export cookie 18190868996380090859 [ 7098.723692] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7098.735052] LustreError: 152359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7099.099704] Lustre: server umount lustre-MDT0001 complete [ 7114.103070] Lustre: server umount lustre-OST0000 complete [ 7141.411234] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 7155.735344] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 7206.760737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7207.033721] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7207.053763] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7207.160108] Lustre: lustre-MDT0000: new disk, initializing [ 7207.240479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7207.262136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7213.125650] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7226.101039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7226.375152] Lustre: 157197:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7226.510774] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7226.521406] Lustre: Skipped 1 previous similar message [ 7226.682793] Lustre: lustre-MDT0001: new disk, initializing [ 7226.916606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7227.053293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7227.072797] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7233.381113] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7243.661858] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7252.684621] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7262.598682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7263.075441] Lustre: lustre-OST0000: new disk, initializing [ 7263.085357] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7263.109322] Lustre: 159168:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7263.280615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7268.481517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7268.493589] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7268.656935] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7269.492889] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7282.758429] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7287.767476] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7288.106684] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7293.964425] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7294.088942] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7299.565389] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7299.566542] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7299.575263] LustreError: Skipped 1 previous similar message [ 7299.575929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7299.652436] Lustre: Skipped 2 previous similar messages [ 7304.680735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7304.699549] Lustre: Skipped 2 previous similar messages [ 7309.801391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7309.809843] Lustre: Skipped 1 previous similar message [ 7311.328177] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7312.150056] Lustre: server umount lustre-OST0000 complete [ 7319.010458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7319.015848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7319.025757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7330.274962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7330.290299] Lustre: Skipped 5 previous similar messages [ 7332.321094] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7332.701933] Lustre: server umount lustre-MDT0000 complete [ 7335.403159] LustreError: 157208:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7335.446641] LustreError: 157208:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7337.043949] LustreError: 157190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786995915 with bad export cookie 18190868996380091818 [ 7337.050287] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7337.055271] LustreError: 157190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7337.366765] Lustre: server umount lustre-MDT0001 complete [ 7342.171868] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 15:45:18 (1786995918) [ 7344.444123] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7347.286755] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 15:45:23 (1786995923) [ 7356.588699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7357.312380] LustreError: 161587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7357.455884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7362.538938] LustreError: 161588:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7363.102114] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7367.657427] LustreError: 161588:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7372.769388] LustreError: 161587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7374.160704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7380.497269] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7390.718107] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7397.660300] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7406.597872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7407.087422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7407.090313] Lustre: Skipped 1 previous similar message [ 7414.210801] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7424.260588] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7437.800944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7437.810527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7437.822249] Lustre: Skipped 3 previous similar messages [ 7437.860462] Lustre: Skipped 2 previous similar messages [ 7442.225108] Lustre: server umount lustre-MDT0000 complete [ 7442.927958] LustreError: 162315:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7442.943026] LustreError: 162315:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7447.549497] LustreError: 161569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786996026 with bad export cookie 18190868996380092700 [ 7447.592714] LustreError: 161569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7447.594684] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7448.037161] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7454.679933] Lustre: server umount lustre-MDT0001 complete [ 7465.880286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7483.176830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7483.372945] LustreError: 165452:0:(llog.c:1655:llog_backup()) MGC192.168.202.141@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7483.381239] Lustre: 165452:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.141@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7490.636407] LustreError: 165457:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7490.655880] LustreError: 165457:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7523.816298] LustreError: 165459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7523.843204] LustreError: 165459:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 22 previous similar messages [ 7544.608333] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7544.621130] Lustre: 165452:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.141@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7585.295185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7590.381349] LustreError: 165459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7590.390121] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7590.400052] LustreError: 165459:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 45 previous similar messages [ 7590.930975] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7601.016577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7613.931051] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7619.055197] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xfc72f2f90ce70275 [ 7619.077824] LustreError: 166210:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.202.141@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (0000000073708b62) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 7619.098927] Lustre: MGC192.168.202.141@tcp: Connection restored to 0@lo (at 0@lo) [ 7624.602243] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7624.699309] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7634.029951] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7636.503616] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7643.885897] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7646.070120] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7670.762150] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7670.763874] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7670.773802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7670.793831] Lustre: Skipped 1 previous similar message [ 7670.830207] Lustre: Skipped 6 previous similar messages [ 7671.487438] Lustre: server umount lustre-MDT0000 complete [ 7676.492384] LustreError: 161568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786996255 with bad export cookie 18190868996380099189 [ 7676.502325] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7676.812672] Lustre: server umount lustre-MDT0001 complete [ 7690.517670] Lustre: server umount lustre-OST0000 complete [ 7717.033871] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 7727.646610] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 7770.686519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7770.916720] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7770.937878] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7771.029158] Lustre: lustre-MDT0000: new disk, initializing [ 7771.198330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7771.201450] Lustre: Skipped 1 previous similar message [ 7771.214366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7776.082520] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7788.472815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7788.633587] Lustre: 170564:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7788.894679] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7794.637216] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7805.517559] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7811.841386] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7819.340617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7819.707454] Lustre: lustre-OST0000: new disk, initializing [ 7819.709975] Lustre: Skipped 1 previous similar message [ 7819.714266] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7819.718374] Lustre: Skipped 2 previous similar messages [ 7819.722827] Lustre: 172530:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7826.481697] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7828.537646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7828.544677] Lustre: Skipped 1 previous similar message [ 7828.551629] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7828.643155] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7836.567364] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7841.563424] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7841.782519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7847.025970] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7847.300515] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7849.443586] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7849.459516] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7849.462557] Lustre: Skipped 1 previous similar message [ 7849.483305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7849.498182] Lustre: Skipped 1 previous similar message [ 7855.481174] Lustre: server umount lustre-OST0000 complete [ 7864.832718] LustreError: 170576:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7864.848398] LustreError: 170576:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 27 previous similar messages [ 7864.919751] Lustre: server umount lustre-MDT0000 complete [ 7871.276258] LustreError: 170556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786996450 with bad export cookie 18190868996380115996 [ 7871.298163] LustreError: 170556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7871.300496] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7872.142861] Lustre: server umount lustre-MDT0001 complete [ 7886.642665] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 15:54:21 (1786996461) [ 7895.101895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7900.872612] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7911.090855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7916.830176] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7924.562703] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7933.344510] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7942.877620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7949.850164] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7958.206141] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7980.475751] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 15:55:56 (1786996556) [ 7986.660552] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7986.662417] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7986.684049] Lustre: Skipped 3 previous similar messages [ 7986.700766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7986.709432] Lustre: Skipped 5 previous similar messages [ 7990.696446] Lustre: server umount lustre-OST0000 complete [ 8001.539904] Lustre: server umount lustre-MDT0000 complete [ 8005.566986] LustreError: 175680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786996584 with bad export cookie 18190868996380116871 [ 8005.568966] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8005.576502] LustreError: 175680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8005.830723] Lustre: server umount lustre-MDT0001 complete [ 8017.495913] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 8020.605777] Key type lgssc unregistered [ 8020.910826] LNet: 178703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8020.938052] LNetError: 178703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8020.955868] LNet: Removed LNI 192.168.202.141@tcp [ 8021.945181] Key type .llcrypt unregistered [ 8021.947813] Key type ._llcrypt unregistered [ 8042.472669] Key type ._llcrypt registered [ 8042.475537] Key type .llcrypt registered [ 8042.565587] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 8044.123713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8044.192342] alg: No test for adler32 (adler32-zlib) [ 8045.394285] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 8045.628928] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 8047.313161] Key type lgssc registered [ 8048.516542] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8057.365143] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8057.379991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8072.992619] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8078.112357] LustreError: 179946:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.141@tcp: failed processing log, type 4: rc = -110 [ 8083.232521] LustreError: 179946:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8083.253624] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8083.265466] LustreError: MGC192.168.202.141@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8083.275212] LustreError: 179946:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8083.285833] LustreError: 179946:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 8083.297391] LustreError: 179946:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8083.418220] Lustre: server umount lustre-OST0000 complete [ 8083.427744] LustreError: 179946:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8092.957413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8094.484419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8100.500567] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8104.932108] LustreError: 180444:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8104.969047] LustreError: 180444:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8110.050591] LustreError: 180445:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8110.159763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8110.624229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8115.252405] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8123.805293] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8129.112527] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8136.799469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8136.976531] LustreError: 182262:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8136.985629] Lustre: 182262:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.141@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8137.155080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8143.598769] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8150.929516] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8154.593371] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8154.595844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8154.603909] Lustre: Skipped 1 previous similar message [ 8154.616455] Lustre: Skipped 2 previous similar messages [ 8155.618625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8155.630051] Lustre: Skipped 1 previous similar message [ 8160.738024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8160.752580] Lustre: Skipped 2 previous similar messages [ 8165.861080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8165.871130] Lustre: Skipped 2 previous similar messages [ 8167.392167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8167.682549] Lustre: server umount lustre-MDT0000 complete [ 8170.996871] LustreError: 180444:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8171.046706] LustreError: 180444:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8172.684546] LustreError: 180425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786996751 with bad export cookie 831656748333854642 [ 8172.690620] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8172.691614] LustreError: 180425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8174.112326] LustreError: 181173:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000008797fd98 x1873801772038912/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8174.406764] Lustre: server umount lustre-MDT0001 complete [ 8189.635904] Lustre: server umount lustre-OST0000 complete [ 8199.796911] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 8210.941826] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 8254.130844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8254.474338] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8254.506544] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8254.594372] Lustre: lustre-MDT0000: new disk, initializing [ 8254.691436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8254.718424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8260.150976] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8271.759226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8271.916386] Lustre: 185789:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8271.986234] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8271.990367] Lustre: Skipped 1 previous similar message [ 8272.086180] Lustre: lustre-MDT0001: new disk, initializing [ 8272.227591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8272.278471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8272.287992] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8277.462872] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8288.559075] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8296.587529] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8304.503970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8304.728160] Lustre: lustre-OST0000: new disk, initializing [ 8304.731887] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8304.736431] Lustre: 187768:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8304.842109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8312.378362] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8314.447367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8314.455453] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8314.569776] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8323.099095] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8328.793373] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8329.165847] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8334.759460] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8335.026661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8340.449731] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8340.454834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8340.475044] Lustre: Skipped 1 previous similar message [ 8340.477395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8340.479326] Lustre: Skipped 1 previous similar message [ 8343.612319] Lustre: server umount lustre-OST0000 complete [ 8349.154432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8349.159993] LustreError: Skipped 1 previous similar message [ 8349.168062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8349.180315] Lustre: Skipped 1 previous similar message [ 8349.191404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8355.427888] Lustre: server umount lustre-MDT0000 complete [ 8355.809727] LustreError: 185800:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8355.824916] LustreError: 185800:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8360.004501] LustreError: 185780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786996938 with bad export cookie 831656748333855356 [ 8360.018570] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8360.023770] LustreError: 185780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8360.419512] Lustre: server umount lustre-MDT0001 complete [ 8374.787315] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 16:02:30 (1786996950) [ 8397.781638] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 8400.686990] Key type lgssc unregistered [ 8400.920941] LNet: 190719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8400.925825] LNetError: 190719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8400.936506] LNet: Removed LNI 192.168.202.141@tcp [ 8401.827144] Key type .llcrypt unregistered [ 8401.831152] Key type ._llcrypt unregistered [ 8428.470231] Key type ._llcrypt registered [ 8428.475940] Key type .llcrypt registered [ 8428.553880] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 8430.006357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8430.056556] alg: No test for adler32 (adler32-zlib) [ 8431.141955] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 8431.510055] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 8433.280450] Key type lgssc registered [ 8434.532217] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8443.587660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8443.605422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8445.134105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8450.047533] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8454.630773] LustreError: 191945:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8454.654708] LustreError: 191945:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8459.754649] LustreError: 191946:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8460.643503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8461.073230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8466.374282] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8476.749435] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8486.520842] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8499.586871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8499.885911] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8499.890920] LustreError: 193812:0:(llog.c:1696:llog_backup()) MGC192.168.202.141@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8499.901437] Lustre: 193812:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.141@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8500.103513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8507.991825] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8517.952316] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8522.727581] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8522.733892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8522.745320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8524.773725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8524.776447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8524.794800] Lustre: Skipped 1 previous similar message [ 8524.807072] Lustre: Skipped 2 previous similar messages [ 8527.034804] Lustre: server umount lustre-MDT0000 complete [ 8529.893450] LustreError: 191946:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8529.918283] LustreError: 191946:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8531.324465] LustreError: 191925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786997110 with bad export cookie 9768171238351138856 [ 8531.344351] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8531.814966] Lustre: server umount lustre-MDT0001 complete [ 8547.442312] Lustre: server umount lustre-OST0000 complete [ 8559.752381] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 8572.849232] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 8622.618034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8622.950206] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8623.000738] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8623.091618] Lustre: lustre-MDT0000: new disk, initializing [ 8623.169672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8623.187632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8629.299526] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8642.374685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8642.509448] Lustre: 197341:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8642.537982] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8642.540222] Lustre: Skipped 1 previous similar message [ 8642.542616] LustreError: 198124:0:(llog.c:1696:llog_backup()) MGC192.168.202.141@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8642.547728] LustreError: 198124:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8642.552235] Lustre: 198124:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.141@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8642.567990] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8642.570989] Lustre: Skipped 1 previous similar message [ 8642.621306] Lustre: lustre-MDT0001: new disk, initializing [ 8642.699230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8642.748431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8642.758711] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8647.722416] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8659.662612] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8666.795993] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8675.497354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8675.835987] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8675.838659] Lustre: Skipped 1 previous similar message [ 8675.841456] LustreError: 199286:0:(llog.c:1696:llog_backup()) MGC192.168.202.141@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8675.852541] LustreError: 199286:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8675.855430] Lustre: 199286:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.141@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8675.863373] Lustre: 199286:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8675.880568] Lustre: lustre-OST0000: new disk, initializing [ 8675.884915] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8675.893564] Lustre: 199310:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8675.984708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8684.434854] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8686.146431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8686.168905] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8686.418671] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8695.758556] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8701.923088] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8702.140785] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8707.318593] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8707.772472] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8712.168394] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8712.169795] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8712.181997] LustreError: Skipped 1 previous similar message [ 8712.183227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8712.203081] Lustre: Skipped 1 previous similar message [ 8712.221521] Lustre: Skipped 1 previous similar message [ 8716.237273] Lustre: server umount lustre-OST0000 complete [ 8722.401479] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8722.408356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8722.422427] Lustre: Skipped 1 previous similar message [ 8722.431664] Lustre: Skipped 1 previous similar message [ 8727.521630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8735.200131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8735.592943] Lustre: server umount lustre-MDT0000 complete [ 8737.768694] LustreError: 198521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8737.786853] LustreError: 198521:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8740.168663] LustreError: 197334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786997318 with bad export cookie 9768171238351139577 [ 8740.179719] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8740.184343] LustreError: 197334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8740.641478] Lustre: server umount lustre-MDT0001 complete [ 8752.106667] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 16:08:48 (1786997328) [ 8770.731719] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 8782.473679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8782.863273] LustreError: 202439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8782.895635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8787.991644] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8793.068568] LustreError: 202440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8793.095250] LustreError: 202440:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8798.148644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8804.223954] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8807.497291] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8816.168740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8816.662360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8816.670369] Lustre: Skipped 1 previous similar message [ 8824.333650] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8834.496617] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8834.688315] Lustre: lustre-OST0001: new disk, initializing [ 8834.693509] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8834.701565] Lustre: 204748:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8842.441693] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8844.436218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8844.446401] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8844.633674] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8856.791030] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8865.564551] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8868.544264] Lustre: DEBUG MARKER: ADD OST3 [ 8875.333262] loop: module loaded [ 8875.428424] loop0: detected capacity change from 0 to 204800000 [ 8875.828119] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8875.851355] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8875.859770] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8883.518873] loop0: detected capacity change from 0 to 204800000 [ 8883.771088] Lustre: DEBUG MARKER: STOP OST3 [ 8891.037184] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8891.359834] Lustre: lustre-OST0003: new disk, initializing [ 8891.376532] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8891.392144] Lustre: 206795:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8891.543970] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8891.550356] Lustre: Skipped 1 previous similar message [ 8893.428120] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8893.585991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8893.596238] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8893.659208] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8894.964483] Lustre: Failing over lustre-OST0003 [ 8895.153351] Lustre: server umount lustre-OST0003 complete [ 8895.462597] LustreError: 206935:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8898.095567] LustreError: 203926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8898.132802] LustreError: 203926:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8898.566954] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8899.759553] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8908.521566] Lustre: DEBUG MARKER: START OST3 again [ 8917.181798] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8917.790754] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8918.542278] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8919.642857] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8919.645451] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8925.046151] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8947.685646] Lustre: Failing over lustre-OST0003 [ 8947.921598] Lustre: server umount lustre-OST0003 complete [ 8948.709091] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8948.728740] Lustre: Skipped 1 previous similar message [ 8948.732888] LustreError: 203926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8948.762461] LustreError: 203926:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 8969.188281] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8969.196158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8969.215312] Lustre: Skipped 2 previous similar messages [ 8969.239367] Lustre: Skipped 6 previous similar messages [ 8975.782969] Lustre: server umount lustre-MDT0000 complete [ 8981.353213] LustreError: 202421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786997560 with bad export cookie 9768171238351140459 [ 8981.384836] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8986.595589] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8986.603734] Lustre: Skipped 6 previous similar messages [ 8988.107905] Lustre: server umount lustre-MDT0001 complete [ 8993.076538] Lustre: server umount lustre-OST0000 complete [ 8997.831118] Lustre: server umount lustre-OST0001 complete [ 9007.468522] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 9018.506453] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 9062.529467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9062.804590] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9062.866212] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9063.013521] Lustre: lustre-MDT0000: new disk, initializing [ 9063.174233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9063.181687] Lustre: Skipped 1 previous similar message [ 9063.221974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9068.742916] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9081.336216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9081.461621] Lustre: 212210:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9081.513091] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9081.515409] Lustre: Skipped 1 previous similar message [ 9081.585736] Lustre: lustre-MDT0001: new disk, initializing [ 9081.726784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9081.740372] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9087.683823] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9099.151548] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9106.298583] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9113.612383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9113.851989] Lustre: lustre-OST0000: new disk, initializing [ 9113.859072] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9113.871987] Lustre: 214179:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9115.625071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9115.650067] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9115.809088] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9121.413064] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9130.671866] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9136.024285] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9136.255386] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9142.089957] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9142.354705] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9146.352519] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9146.371837] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9146.375647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9146.389770] Lustre: Skipped 4 previous similar messages [ 9151.462973] LustreError: 214278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9151.473755] LustreError: 214278:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [ 9151.596548] Lustre: server umount lustre-OST0000 complete [ 9162.869631] Lustre: server umount lustre-MDT0000 complete [ 9166.978377] LustreError: 212202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786997745 with bad export cookie 9768171238351142755 [ 9166.985459] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9167.004945] LustreError: 212202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9167.598674] Lustre: server umount lustre-MDT0001 complete [ 9171.384088] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 16:15:47 (1786997747) [ 9191.208966] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 9200.597289] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 9244.014393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9244.306831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9244.359165] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9244.465698] Lustre: lustre-MDT0000: new disk, initializing [ 9244.574949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9249.683821] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9261.717838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9261.892681] Lustre: 218793:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9262.254313] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9267.495896] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9278.062392] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9284.279580] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9292.164769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9292.465988] Lustre: 220762:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9294.364836] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9300.113973] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9310.161889] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9316.002821] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9316.296724] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9322.070723] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9322.311944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9325.025393] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9325.028782] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9325.047197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9325.047206] Lustre: Skipped 5 previous similar messages [ 9325.083093] Lustre: Skipped 3 previous similar messages [ 9330.622388] Lustre: server umount lustre-OST0000 complete [ 9349.088186] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9349.363692] Lustre: server umount lustre-MDT0000 complete [ 9350.634844] LustreError: 218804:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9350.663939] LustreError: 218804:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9353.826616] LustreError: 218783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786997932 with bad export cookie 9768171238351143637 [ 9353.842169] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9353.849472] LustreError: 218783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9354.167334] Lustre: server umount lustre-MDT0001 complete [ 9364.623561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9365.838050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9365.842658] Lustre: Skipped 5 previous similar messages [ 9371.195107] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9381.420926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9386.929490] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9395.160979] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9401.151302] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9409.086365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9417.364116] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9427.098431] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9440.752759] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 9444.693182] Lustre: server umount lustre-OST0000 complete [ 9456.510277] Lustre: server umount lustre-MDT0000 complete [ 9461.518692] LustreError: 225861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786998040 with bad export cookie 9768171238351144519 [ 9461.528665] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9461.538299] LustreError: 225861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9462.141500] Lustre: server umount lustre-MDT0001 complete [ 9568.252375] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 9570.560454] Key type lgssc unregistered [ 9570.890328] LNet: 226650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9570.896070] LNetError: 226650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9570.916446] LNet: Removed LNI 192.168.202.141@tcp [ 9571.624173] Key type .llcrypt unregistered [ 9571.626502] Key type ._llcrypt unregistered [ 9595.252631] Key type ._llcrypt registered [ 9595.254691] Key type .llcrypt registered [ 9595.340182] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 9609.975988] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 9611.268652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9611.298768] alg: No test for adler32 (adler32-zlib) [ 9612.253812] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 9612.410969] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 9614.027461] Key type lgssc registered [ 9614.951919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9638.726612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9638.767743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9639.958340] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9639.976559] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9640.040253] Lustre: lustre-MDT0000: new disk, initializing [ 9640.098059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9640.114717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9643.299535] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9651.317640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9651.410755] Lustre: 230001:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9651.462935] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9651.465875] Lustre: Skipped 1 previous similar message [ 9651.533374] Lustre: lustre-MDT0001: new disk, initializing [ 9651.620440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9651.673813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9651.683478] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9655.723767] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9662.930691] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9667.418747] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9672.297662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9672.539430] Lustre: lustre-OST0000: new disk, initializing [ 9672.542427] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9672.547377] Lustre: 231967:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9672.632512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9677.597901] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9681.030834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9681.037123] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9681.097160] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9685.154747] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9688.698171] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9688.836332] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9693.117416] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9693.307278] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9696.739856] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9696.753113] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9696.768197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9696.791429] Lustre: Skipped 1 previous similar message [ 9701.481579] Lustre: server umount lustre-OST0000 complete [ 9706.979381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9706.984131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9706.986292] Lustre: Skipped 1 previous similar message [ 9706.994589] Lustre: Skipped 1 previous similar message [ 9711.461604] Lustre: server umount lustre-MDT0000 complete [ 9714.561527] LustreError: 230796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786998293 with bad export cookie 10366893880684143729 [ 9714.568859] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9714.572213] LustreError: 230796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9714.857082] Lustre: server umount lustre-MDT0001 complete [ 9729.368738] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 9738.347173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9738.613324] LustreError: 234898:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9738.621746] LustreError: 234898:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9738.676036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9742.376497] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9743.843213] LustreError: 234899:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9748.964624] LustreError: 234898:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9749.143877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9749.388517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9752.895513] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9755.024294] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9759.619819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9759.724567] LustreError: 236380:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9759.913732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9764.170522] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9771.471026] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9771.567531] Lustre: lustre-OST0001: new disk, initializing [ 9771.570190] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9771.574178] Lustre: 237204:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9776.468254] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9779.758579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9779.766698] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9779.855976] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9786.438556] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9792.006539] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9801.179031] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 16:26:18 (1786998378) [ 9805.807464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9805.810826] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9805.819794] LustreError: Skipped 1 previous similar message [ 9805.823150] Lustre: Skipped 1 previous similar message [ 9805.826806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9805.832888] Lustre: Skipped 3 previous similar messages [ 9809.067484] Lustre: server umount lustre-MDT0000 complete [ 9810.927638] LustreError: 234917:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9810.946737] LustreError: 234917:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9813.612309] LustreError: 236030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786998392 with bad export cookie 10366893880684144611 [ 9813.619432] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9813.622184] LustreError: 236030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9814.897224] Lustre: server umount lustre-MDT0001 complete [ 9831.341431] Lustre: server umount lustre-OST0000 complete [ 9832.040444] Lustre: 228231:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786998394/real 1786998394] req@ffff9460bc9cb480 x1873803415225728/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786998410 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9832.082682] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9832.102765] Lustre: Skipped 2 previous similar messages [ 9833.504458] Lustre: 228228:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786998396/real 1786998396] req@ffff9460bc9c9f80 x1873803415225984/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786998412 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9836.530554] Lustre: 228230:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786998399/real 1786998399] req@ffff9460bc9ca680 x1873803415226240/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786998415 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9837.930860] Lustre: server umount lustre-OST0001 complete [ 9853.796254] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 9869.740885] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 9919.879568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9919.999959] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9920.018566] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9920.061686] Lustre: lustre-MDT0000: new disk, initializing [ 9920.144504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9920.149296] Lustre: Skipped 1 previous similar message [ 9920.160145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9925.815215] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9940.292764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9940.541059] Lustre: 241170:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9940.694650] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9940.704347] Lustre: Skipped 1 previous similar message [ 9940.898293] Lustre: lustre-MDT0001: new disk, initializing [ 9941.150897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9941.163821] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9946.213812] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9957.748334] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9964.287408] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9972.586191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9973.078904] Lustre: lustre-OST0000: new disk, initializing [ 9973.082345] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9973.090577] Lustre: 243142:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9973.247514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9973.250345] Lustre: Skipped 1 previous similar message [ 9974.354029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9974.364685] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9974.587150] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9980.824526] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9992.492929] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9998.773373] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9998.998314] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10004.941859] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10005.190602] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10010.595422] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10010.599212] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10010.609317] LustreError: Skipped 1 previous similar message [10010.619630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10010.638495] Lustre: Skipped 1 previous similar message [10010.663427] Lustre: Skipped 1 previous similar message [10013.454716] Lustre: server umount lustre-OST0000 complete [10020.835253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10020.859057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10020.861280] Lustre: Skipped 1 previous similar message [10024.498195] Lustre: server umount lustre-MDT0000 complete [10025.953870] LustreError: 241180:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10025.960243] LustreError: 241180:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10028.657621] LustreError: 241163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786998607 with bad export cookie 10366893880684146746 [10028.664860] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10028.675882] LustreError: 241163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10029.217890] Lustre: server umount lustre-MDT0001 complete [10039.828706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10040.427306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10045.932252] LustreError: 245360:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10045.955781] LustreError: 245360:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10046.027257] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10056.786480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10063.244690] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10072.438706] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10080.002250] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10087.791301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10096.013645] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10107.167374] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10255.844570] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10255.856367] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10255.868596] Lustre: Skipped 1 previous similar message [10255.876877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10269.152514] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10269.325186] Lustre: server umount lustre-OST0000 complete [10274.797904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10274.800890] Lustre: Skipped 4 previous similar messages [10279.900472] Lustre: server umount lustre-MDT0000 complete [10279.907197] LustreError: 245360:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10279.915441] LustreError: 245360:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10286.330175] LustreError: 245340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786998865 with bad export cookie 10366893880684147628 [10286.344897] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10286.369767] LustreError: 245340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10287.267602] Lustre: server umount lustre-MDT0001 complete [10378.103671] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [10381.737423] Key type lgssc unregistered [10382.067761] LNet: 248974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10382.074438] LNetError: 248974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10382.089628] LNet: Removed LNI 192.168.202.141@tcp [10383.359145] Key type .llcrypt unregistered [10383.361389] Key type ._llcrypt unregistered [10419.740076] Key type ._llcrypt registered [10419.743813] Key type .llcrypt registered [10420.022114] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [10439.767411] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [10441.022474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10441.247423] alg: No test for adler32 (adler32-zlib) [10442.497244] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [10442.801602] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [10444.562330] Key type lgssc registered [10446.006625] Lustre: Echo OBD driver; http://www.lustre.org/ [10503.127172] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [10515.189761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10515.225564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10516.671816] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10516.749922] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10516.878615] Lustre: lustre-MDT0000: new disk, initializing [10516.950378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10516.965905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10521.980362] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10534.258287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10534.442717] Lustre: 253048:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10534.498638] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10534.502761] Lustre: Skipped 1 previous similar message [10534.604625] Lustre: lustre-MDT0001: new disk, initializing [10534.694778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10534.735270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10534.758291] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10539.640272] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10545.165074] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10553.460163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10553.955759] Lustre: lustre-OST0000: new disk, initializing [10553.970209] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10553.975260] Lustre: 254680:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10554.135761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10559.594332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10559.608220] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10559.777093] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10561.719322] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10573.973643] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10574.177313] Lustre: lustre-OST0001: new disk, initializing [10574.181126] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10574.187028] Lustre: 255552:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10574.314538] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10580.605867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10580.618681] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10580.682087] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10581.121221] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10593.730275] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10601.547701] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10616.803983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10616.812965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10616.825830] Lustre: Skipped 1 previous similar message [10616.838698] Lustre: Skipped 2 previous similar messages [10620.018110] Lustre: server umount lustre-MDT0000 complete [10623.913523] LustreError: 256379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786999202 with bad export cookie 18252434912666098072 [10623.921498] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10624.304537] Lustre: server umount lustre-MDT0001 complete [10638.810966] Lustre: server umount lustre-OST0000 complete [10642.913520] Lustre: 250552:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786999205/real 1786999205] req@ffff94608365ed80 x1873804285523200/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786999221 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10642.954197] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10642.967586] Lustre: Skipped 2 previous similar messages [10643.428859] Lustre: server umount lustre-OST0001 complete [10659.542458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10670.698288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10680.976977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10691.399270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10711.552802] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [10723.967742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10723.977314] Lustre: lustre-MDT0000: reset Object Index mappings [10724.273427] LustreError: 260564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10724.291186] LustreError: 260564:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10724.380467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10729.151625] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10729.444569] LustreError: 260565:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10734.564247] LustreError: 260564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10738.717506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10738.749310] Lustre: lustre-MDT0001: reset Object Index mappings [10739.328322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10745.435594] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10748.935931] Lustre: 261707:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10757.107986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10757.583893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10764.792265] LustreError: 262061:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10764.860864] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10769.920955] LustreError: 262062:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10769.968777] LustreError: 262062:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10769.991792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10777.827700] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10783.768126] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10789.010581] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10799.185215] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10803.690273] Lustre: 263578:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10810.853557] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10810.870265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10810.897854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10814.435069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10814.446175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10814.465949] Lustre: Skipped 2 previous similar messages [10815.860759] Lustre: server umount lustre-MDT0000 complete [10819.567402] LustreError: 261518:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10819.586786] LustreError: 261518:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10819.925471] LustreError: 260545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786999398 with bad export cookie 18252434912666104855 [10819.937990] LustreError: 260545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10819.952730] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10820.335793] Lustre: server umount lustre-MDT0001 complete [10834.290194] Lustre: server umount lustre-OST0000 complete [10847.641899] Lustre: server umount lustre-OST0001 complete [10867.855641] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 16:44:04 (1786999444) [10884.511074] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [10893.965329] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local