[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 505831383 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001022] APIC: Switch to symmetric I/O mode setup [ 0.002161] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.005010] kvm-guest: setup PV IPIs [ 0.008494] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010010] pid_max: default: 32768 minimum: 301 [ 0.011243] LSM: Security Framework initializing [ 0.012059] Yama: becoming mindful. [ 0.013055] SELinux: Initializing. [ 0.015044] *** VALIDATE selinux *** [ 0.022809] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027748] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028151] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030112] *** VALIDATE tmpfs *** [ 0.031435] *** VALIDATE proc *** [ 0.032249] *** VALIDATE cgroup *** [ 0.033008] *** VALIDATE cgroup2 *** [ 0.034227] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035275] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037029] Spectre V2 : User space: Vulnerable [ 0.038008] Speculative Store Bypass: Vulnerable [ 0.041466] debug: unmapping init [mem 0xffffffff97c59000-0xffffffff97c60fff] [ 0.044000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044482] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045020] ... version: 2 [ 0.046009] ... bit width: 48 [ 0.047023] ... generic registers: 4 [ 0.048014] ... value mask: 0000ffffffffffff [ 0.049014] ... max period: 00007fffffffffff [ 0.050011] ... fixed-purpose events: 3 [ 0.051012] ... event mask: 000000070000000f [ 0.052281] rcu: Hierarchical SRCU implementation. [ 0.054465] smp: Bringing up secondary CPUs ... [ 0.055598] x86: Booting SMP configuration: [ 0.056032] .... node #0, CPUs: #1 #2 #3 [ 0.082077] smp: Brought up 1 node, 4 CPUs [ 0.084012] smpboot: Max logical packages: 1 [ 0.085019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.161314] node 0 deferred pages initialised in 74ms [ 0.165011] devtmpfs: initialized [ 0.166232] x86/mm: Memory block size: 128MB [ 0.168961] gcov: version magic: 0x41383552 [ 0.170163] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.171075] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.172220] pinctrl core: initialized pinctrl subsystem [ 0.173169] [ 0.173701] ************************************************************* [ 0.174016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.175011] ** ** [ 0.176014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.177014] ** ** [ 0.178010] ** This means that this kernel is built to expose internal ** [ 0.179013] ** IOMMU data structures, which may compromise security on ** [ 0.180013] ** your system. ** [ 0.181013] ** ** [ 0.182013] ** If you see this message and you are not debugging the ** [ 0.183012] ** kernel, report this immediately to your vendor! ** [ 0.184011] ** ** [ 0.185012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.186014] ************************************************************* [ 0.187709] NET: Registered protocol family 16 [ 0.188502] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.189051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.190057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.191438] cpuidle: using governor menu [ 0.194254] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.196516] PCI: Using configuration type 1 for base access [ 0.198202] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.207099] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.210129] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.214053] cryptd: max_cpu_qlen set to 1000 [ 0.217201] ACPI: Added _OSI(Module Device) [ 0.218012] ACPI: Added _OSI(Processor Device) [ 0.220032] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.222012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.226910] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.234650] ACPI: Interpreter enabled [ 0.236054] ACPI: PM: (supports S0 S3 S4 S5) [ 0.238021] ACPI: Using IOAPIC for interrupt routing [ 0.240092] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.243446] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.254419] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.257037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.260071] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.264077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.268921] acpiphp: Slot [2] registered [ 0.270101] acpiphp: Slot [5] registered [ 0.272098] acpiphp: Slot [6] registered [ 0.274113] acpiphp: Slot [7] registered [ 0.275255] acpiphp: Slot [8] registered [ 0.277088] acpiphp: Slot [9] registered [ 0.279097] acpiphp: Slot [10] registered [ 0.281090] acpiphp: Slot [3] registered [ 0.283093] acpiphp: Slot [4] registered [ 0.284218] acpiphp: Slot [11] registered [ 0.286094] acpiphp: Slot [12] registered [ 0.288147] acpiphp: Slot [13] registered [ 0.289147] acpiphp: Slot [14] registered [ 0.291106] acpiphp: Slot [15] registered [ 0.293089] acpiphp: Slot [16] registered [ 0.294117] acpiphp: Slot [17] registered [ 0.296091] acpiphp: Slot [18] registered [ 0.298109] acpiphp: Slot [19] registered [ 0.299107] acpiphp: Slot [20] registered [ 0.301098] acpiphp: Slot [21] registered [ 0.303108] acpiphp: Slot [22] registered [ 0.304137] acpiphp: Slot [23] registered [ 0.306107] acpiphp: Slot [24] registered [ 0.308181] acpiphp: Slot [25] registered [ 0.310118] acpiphp: Slot [26] registered [ 0.312105] acpiphp: Slot [27] registered [ 0.314109] acpiphp: Slot [28] registered [ 0.316126] acpiphp: Slot [29] registered [ 0.317091] acpiphp: Slot [30] registered [ 0.319099] acpiphp: Slot [31] registered [ 0.321062] PCI host bridge to bus 0000:00 [ 0.322018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.325025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.327024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.330024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.333025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.336024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.338162] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.341095] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.345362] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.356872] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.362057] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.365017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.367014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.370017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.373544] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.379071] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.382044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.385888] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.392017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.407016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.411902] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.416801] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.422014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.428012] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.447015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.454915] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.467016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.477015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.507016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.520371] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.529013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.540013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.566015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.580033] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.593014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.606021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.638021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.650399] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.661014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.674015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.703015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.723330] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.739018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.753013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.780017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.797116] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.800397] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.803410] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.806399] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.809205] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.814033] iommu: Default domain type: Passthrough [ 0.817409] SCSI subsystem initialized [ 0.819138] ACPI: bus type USB registered [ 0.821104] usbcore: registered new interface driver usbfs [ 0.823113] usbcore: registered new interface driver hub [ 0.826159] usbcore: registered new device driver usb [ 0.828187] pps_core: LinuxPPS API ver. 1 registered [ 0.830011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.834135] PTP clock support registered [ 0.836142] EDAC MC: Ver: 3.0.0 [ 0.837665] PCI: Using ACPI for IRQ routing [ 0.840999] NetLabel: Initializing [ 0.842010] NetLabel: domain hash size = 128 [ 0.844010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.846079] NetLabel: unlabeled traffic allowed by default [ 0.849105] vgaarb: loaded [ 0.850295] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.853021] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.858798] clocksource: Switched to clocksource kvm-clock [ 0.963471] VFS: Disk quotas dquot_6.6.0 [ 0.965279] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.967535] *** VALIDATE ramfs *** [ 0.968594] *** VALIDATE hugetlbfs *** [ 0.970132] pnp: PnP ACPI init [ 0.972382] pnp: PnP ACPI: found 6 devices [ 0.993643] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.996369] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.998520] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.002926] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.005623] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.008376] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.011498] NET: Registered protocol family 2 [ 1.014625] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.020888] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.025398] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.033725] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.039328] TCP: Hash tables configured (established 65536 bind 65536) [ 1.042114] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.045174] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.047990] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.050921] NET: Registered protocol family 1 [ 1.053938] RPC: Registered named UNIX socket transport module. [ 1.055912] RPC: Registered udp transport module. [ 1.057226] RPC: Registered tcp transport module. [ 1.058632] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.060655] NET: Registered protocol family 44 [ 1.062120] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.064081] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.065932] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.068040] PCI: CLS 0 bytes, default 64 [ 1.069483] Unpacking initramfs... [ 2.470848] debug: unmapping init [mem 0xffffa092fcc54000-0xffffa092fffbffff] [ 2.474626] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.476772] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.479228] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.987210] Initialise system trusted keyrings [ 2.989030] Key type blacklist registered [ 2.991077] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.000942] zbud: loaded [ 3.003824] *** VALIDATE nfs *** [ 3.005091] *** VALIDATE nfs4 *** [ 3.006754] pstore: using deflate compression [ 3.011038] Platform Keyring initialized [ 3.117471] NET: Registered protocol family 38 [ 3.119388] Key type asymmetric registered [ 3.120938] Asymmetric key parser 'x509' registered [ 3.123174] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.126797] io scheduler mq-deadline registered [ 3.128564] io scheduler kyber registered [ 3.130328] io scheduler bfq registered [ 3.132171] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.135315] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.138258] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.140915] ACPI: Power Button [PWRF] [ 3.146348] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.155108] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.182295] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.193898] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.216765] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.248633] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.279968] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.285101] Non-volatile memory driver v1.3 [ 3.287239] Linux agpgart interface v0.103 [ 3.337424] virtio_blk virtio1: [vda] 135296 512-byte logical blocks (69.3 MB/66.1 MiB) [ 3.341541] vda: detected capacity change from 0 to 69271552 [ 3.361572] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.365116] vdb: detected capacity change from 0 to 1073741824 [ 3.383518] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.386446] vdc: detected capacity change from 0 to 2621440000 [ 3.405147] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.408372] vdd: detected capacity change from 0 to 2621440000 [ 3.424547] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.426388] vde: detected capacity change from 0 to 4294967296 [ 3.443987] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.446560] vdf: detected capacity change from 0 to 4294967296 [ 3.457374] libphy: Fixed MDIO Bus: probed [ 3.463492] usbcore: registered new interface driver usbserial_generic [ 3.467083] usbserial: USB Serial support registered for generic [ 3.470629] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.476609] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.479392] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.482515] mousedev: PS/2 mouse device common for all mice [ 3.490431] rtc_cmos 00:05: RTC can wake from S4 [ 3.495568] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.495787] rtc_cmos 00:05: registered as rtc0 [ 3.501629] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.503124] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.504472] intel_pstate: CPU model not supported [ 3.509923] hid: raw HID events driver (C) Jiri Kosina [ 3.511354] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.513140] usbcore: registered new interface driver usbhid [ 3.519201] usbhid: USB HID core driver [ 3.521508] drop_monitor: Initializing network drop monitor service [ 3.525380] Initializing XFRM netlink socket [ 3.528064] NET: Registered protocol family 10 [ 3.531399] Segment Routing with IPv6 [ 3.532648] NET: Registered protocol family 17 [ 3.534840] mpls_gso: MPLS GSO support [ 3.539801] RAS: Correctable Errors collector initialized. [ 3.542165] AVX version of gcm_enc/dec engaged. [ 3.544041] AES CTR mode by8 optimization enabled [ 3.618715] sched_clock: Marking stable (3618690338, 0)->(4568404644, -949714306) [ 3.623155] registered taskstats version 1 [ 3.625273] Loading compiled-in X.509 certificates [ 3.627194] zswap: loaded using pool lzo/zbud [ 3.649628] Key type big_key registered [ 3.668096] Key type encrypted registered [ 3.669688] ima: No TPM chip found, activating TPM-bypass! [ 3.672533] ima: Allocated hash algorithm: sha1 [ 3.674606] ima: No architecture policies found [ 3.676743] evm: Initialising EVM extended attributes: [ 3.679619] evm: security.selinux [ 3.681130] evm: security.ima [ 3.682256] evm: security.capability [ 3.683927] evm: HMAC attrs: 0x1 [ 3.686323] rtc_cmos 00:05: setting system clock to 2026-05-28 17:45:34 UTC (1779990334) [ 3.692979] debug: unmapping init [mem 0xffffffff98c03000-0xffffffff98dfffff] [ 3.696661] debug: unmapping init [mem 0xffffffff97982000-0xffffffff97c58fff] [ 3.706058] Write protecting the kernel read-only data: 28672k [ 3.710283] debug: unmapping init [mem 0xffffffff96003000-0xffffffff961fffff] [ 3.713544] debug: unmapping init [mem 0xffffffff96914000-0xffffffff969fffff] [ 3.753327] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.760670] systemd[1]: Detected virtualization kvm. [ 3.762056] systemd[1]: Detected architecture x86-64. [ 3.763869] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.789032] systemd[1]: No hostname configured. [ 3.791081] systemd[1]: Set hostname to . [ 3.793473] random: systemd: uninitialized urandom read (16 bytes read) [ 3.796164] systemd[1]: Initializing machine ID from random generator. [ 3.865791] random: ln: uninitialized urandom read (6 bytes read) [ 3.955759] random: systemd: uninitialized urandom read (16 bytes read) [ 3.958947] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.969406] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.977640] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.581773] device-mapper: uevent: version 1.0.3 [ 4.583901] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System.[ 5.319848] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.486619] virtio_net virtio0 ens2: renamed from eth0 [ 5.519689] scsi host0: ata_piix [ 5.651045] scsi host1: ata_piix [ 5.652354] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.654938] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.165368] dracut-initqueue[569]: RTNETLINK answers: File exists [ 10.189333] random: crng init done [ 10.190602] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.680348] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.018199] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.293211] SELinux: Disabled at runtime. [ 12.353958] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.366363] systemd[1]: Detected virtualization kvm. [ 12.368239] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.899959] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.902625] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.906506] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.910338] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.913548] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.923733] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.938097] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPC[ 13.109708] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS bind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.418096] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.667806] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.670421] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.827636] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.836915] EDAC sbridge: Ver: 1.1.2 [ 15.727781] Key type dns_resolver registered [ 16.052771] NFS: Registering the id_resolver key type [ 16.054696] Key type id_resolver registered [ 16.056418] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Login Service... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg339-server login: [ 47.473480] libcfs: loading out-of-tree module taints kernel. [ 47.599361] Key type ._llcrypt registered [ 47.604466] Key type .llcrypt registered [ 47.777948] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 73.042745] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 75.266255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 75.281869] alg: No test for adler32 (adler32-zlib) [ 77.026682] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 78.375798] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 80.375176] Key type lgssc registered [ 82.374847] Lustre: Echo OBD driver; http://www.lustre.org/ [ 99.373810] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 122.375309] hrtimer: interrupt took 15808566 ns [ 147.928883] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 162.738258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 162.785033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 164.180312] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 164.259154] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 164.448176] Lustre: lustre-MDT0000: new disk, initializing [ 164.580474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 164.609330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 169.112205] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 185.310569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 185.541269] Lustre: 6493:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 185.600302] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 185.608956] Lustre: Skipped 1 previous similar message [ 185.834887] Lustre: lustre-MDT0001: new disk, initializing [ 185.982043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 186.043462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 186.065165] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 192.428042] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 198.206663] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 210.178531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 210.679833] Lustre: lustre-OST0000: new disk, initializing [ 210.685066] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 210.757472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 217.151345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 217.175285] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 217.239277] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 219.495303] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 236.032758] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 236.202210] Lustre: lustre-OST0001: new disk, initializing [ 236.209293] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 236.288922] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 242.794285] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 245.814766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 245.823481] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 245.925157] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 254.768624] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 261.438923] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 273.935969] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing check_logdir /tmp/testlogs/ [ 278.864558] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing yml_node [ 283.974548] Lustre: DEBUG MARKER: Client: 2.17.50.143 [ 286.375953] Lustre: DEBUG MARKER: MDS: 2.17.50.143 [ 289.229485] Lustre: DEBUG MARKER: OSS: 2.17.50.143 [ 290.975566] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu May 28 13:50:19 EDT 2026 [ 313.306731] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 315.172485] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 332.773045] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 332.779902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 332.815226] Lustre: Skipped 3 previous similar messages [ 332.821751] Lustre: Skipped 2 previous similar messages [ 337.998632] Lustre: server umount lustre-MDT0000 complete [ 342.286464] LustreError: 6484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779990673 with bad export cookie 13725300728974214268 [ 342.288432] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 342.295024] LustreError: 6484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 343.012879] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 343.013169] LustreError: 6499:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.020322] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 343.020332] Lustre: Skipped 1 previous similar message [ 343.025818] Lustre: Skipped 1 previous similar message [ 343.062094] LustreError: 6499:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 347.040279] LustreError: 6499:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 347.040548] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 347.069540] Lustre: Skipped 2 previous similar messages [ 348.640765] Lustre: server umount lustre-MDT0001 complete [ 352.654859] Lustre: server umount lustre-OST0000 complete [ 356.702301] Lustre: server umount lustre-OST0001 complete [ 364.242208] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 374.967764] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 416.901643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 417.261843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 417.312375] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 417.418756] Lustre: lustre-MDT0000: new disk, initializing [ 417.548319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 417.565161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 422.402612] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 433.577532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 433.792662] Lustre: 14972:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 433.953057] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 433.965929] Lustre: Skipped 1 previous similar message [ 434.165167] Lustre: lustre-MDT0001: new disk, initializing [ 434.377533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 434.449850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 434.460730] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 440.233320] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 448.665594] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 453.079613] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 459.585754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 459.762485] Lustre: lustre-OST0000: new disk, initializing [ 459.768773] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 459.876555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 461.822995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 461.832270] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 461.954569] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 466.686596] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 475.404561] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 480.313250] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 480.574644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 485.874477] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 486.226563] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 492.536640] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 492.537792] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 492.545787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 492.558113] Lustre: Skipped 1 previous similar message [ 492.578492] Lustre: Skipped 1 previous similar message [ 497.635648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 497.647191] Lustre: Skipped 1 previous similar message [ 503.263796] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 503.490548] Lustre: server umount lustre-OST0000 complete [ 507.872468] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 507.877630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 507.886129] Lustre: Skipped 1 previous similar message [ 507.893479] Lustre: Skipped 2 previous similar messages [ 513.588123] Lustre: server umount lustre-MDT0000 complete [ 518.029254] LustreError: 14962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779990848 with bad export cookie 13725300728974215801 [ 518.034058] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 518.047223] LustreError: 14962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 518.432884] Lustre: server umount lustre-MDT0001 complete [ 524.148639] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 13:54:12 (1779990852) [ 544.093165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 544.482817] LustreError: 19445:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.502408] LustreError: 19445:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 544.544843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 549.051950] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 549.858706] LustreError: 19445:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 554.981306] LustreError: 19445:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 557.914693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 563.468342] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 569.865574] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 574.502794] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 577.368642] Lustre: Failing over lustre-MDT0000 [ 577.676075] Lustre: server umount lustre-MDT0000 complete [ 579.040209] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 579.048246] LustreError: Skipped 1 previous similar message [ 579.057928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 579.067940] LustreError: 19444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 595.232094] Lustre: 3641:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779990910/real 1779990910] req@ffffa09242ef9500 x1866455223274240/t0(0) o400->MGC192.168.203.139@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1779990926 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 595.269681] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 595.297566] LustreError: 19444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 595.319393] LustreError: 19444:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 598.935723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 606.107760] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 611.304616] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 611.319423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 611.319966] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 611.336771] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 620.038592] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 625.634607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 625.636060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 625.660079] Lustre: Skipped 2 previous similar messages [ 625.673544] Lustre: Skipped 4 previous similar messages [ 630.301751] Lustre: server umount lustre-MDT0000 complete [ 633.831497] LustreError: 19445:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 633.852478] LustreError: 19445:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 634.215296] LustreError: 19425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779990965 with bad export cookie 13725300728974217215 [ 634.226355] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 634.238327] LustreError: 19425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 634.763765] Lustre: server umount lustre-MDT0001 complete [ 644.054298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 644.625894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 644.629604] Lustre: Skipped 2 previous similar messages [ 649.557535] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 659.632640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 664.524826] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 671.588185] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 676.438347] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 679.017992] Lustre: Failing over lustre-MDT0000 [ 679.319472] Lustre: server umount lustre-MDT0000 complete [ 680.418875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 680.436379] Lustre: Skipped 1 previous similar message [ 696.672145] Lustre: 3642:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779991011/real 1779991011] req@ffffa09243422680 x1866455223303424/t0(0) o400->MGC192.168.203.139@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1779991027 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 696.739401] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 698.171231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 701.922668] LustreError: 22441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 701.945306] LustreError: 22441:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 707.454385] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 711.609432] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 712.676509] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 712.681259] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 712.703153] Lustre: Skipped 1 previous similar message [ 712.717857] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 719.678689] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 728.034359] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 728.054345] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 728.112374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 728.124663] Lustre: Skipped 2 previous similar messages [ 739.296117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 739.540778] Lustre: server umount lustre-MDT0000 complete [ 744.480150] LustreError: 22424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779991075 with bad export cookie 13725300728974218153 [ 744.494907] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 744.501738] LustreError: 22424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 744.854606] Lustre: server umount lustre-MDT0001 complete [ 754.105679] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 13:58:02 (1779991082) [ 773.899566] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 782.765419] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 823.880550] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 835.517668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 835.774883] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 835.830166] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 835.929899] Lustre: lustre-MDT0000: new disk, initializing [ 836.014063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 836.020864] Lustre: Skipped 2 previous similar messages [ 836.034165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 841.651619] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 852.882202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 852.997617] Lustre: 28428:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 853.289595] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 857.166418] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 866.902696] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 872.728678] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 880.478853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 881.952248] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 881.979510] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 882.006612] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 882.015871] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 886.078676] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 894.092039] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 900.466534] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 923.106121] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 923.109281] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 923.109624] LustreError: Skipped 1 previous similar message [ 923.117063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 923.121564] Lustre: Skipped 2 previous similar messages [ 923.124417] Lustre: Skipped 4 previous similar messages [ 927.971409] Lustre: server umount lustre-OST0000 complete [ 947.167445] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 947.381177] Lustre: server umount lustre-MDT0000 complete [ 948.706704] LustreError: 28439:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 948.733198] LustreError: 28439:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 951.103087] LustreError: 30169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779991281 with bad export cookie 13725300728974218559 [ 951.103945] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 951.110409] LustreError: 30169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 951.547715] Lustre: server umount lustre-MDT0001 complete [ 961.545381] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 964.123896] Key type lgssc unregistered [ 964.417686] LNet: 32034:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 964.429627] LNetError: 32034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 965.481726] LNet: Removed LNI 192.168.203.139@tcp [ 966.332193] Key type .llcrypt unregistered [ 966.334432] Key type ._llcrypt unregistered [ 988.865907] Key type ._llcrypt registered [ 988.868450] Key type .llcrypt registered [ 988.988379] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 989.926290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 989.947216] alg: No test for adler32 (adler32-zlib) [ 991.077304] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 991.279866] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 992.951173] Key type lgssc registered [ 994.017909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1001.358490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1001.394286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1002.980474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1006.857196] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1012.201537] LustreError: 33433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1012.222676] LustreError: 33433:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1015.281375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1015.701658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1019.563805] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1025.703594] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1030.178657] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1037.578379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1038.058534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1041.135880] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1044.707857] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1047.337685] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1050.520328] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1064.381516] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 14:03:12 (1779991392) [ 1066.780346] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1068.743215] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1076.379505] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 14:03:24 (1779991404) [ 1085.709412] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 14:03:33 (1779991413) [ 1086.881373] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1097.360645] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1097.367803] Lustre: Skipped 2 previous similar messages [ 1100.167657] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 14:03:47 (1779991427) [ 1102.892420] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1102.900625] Lustre: Skipped 2 previous similar messages [ 1112.516522] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1112.528765] Lustre: Skipped 4 previous similar messages [ 1126.396298] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 14:04:14 (1779991454) [ 1129.368879] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1132.826545] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1140.827683] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 14:04:29 (1779991469) [ 1143.002858] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1143.008245] Lustre: Skipped 6 previous similar messages [ 1156.355600] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 14:04:44 (1779991484) [ 1173.478873] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1173.517069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1173.524830] Lustre: Skipped 1 previous similar message [ 1178.598647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1178.607992] Lustre: Skipped 1 previous similar message [ 1183.722152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1183.727869] Lustre: Skipped 1 previous similar message [ 1185.247220] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1185.450133] Lustre: server umount lustre-OST0000 complete [ 1189.856295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1189.865819] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1189.875925] Lustre: Skipped 1 previous similar message [ 1189.885931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1193.956450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1193.960229] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1195.378121] Lustre: server umount lustre-MDT0000 complete [ 1198.715292] LustreError: 36659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779991529 with bad export cookie 14630411371685289403 [ 1198.731308] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1198.735485] LustreError: 36659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1198.991662] Lustre: server umount lustre-MDT0001 complete [ 1210.375260] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1213.065185] Key type lgssc unregistered [ 1213.395636] LNet: 40942:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1213.401932] LNetError: 40942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1213.418951] LNet: Removed LNI 192.168.203.139@tcp [ 1214.208426] Key type .llcrypt unregistered [ 1214.210481] Key type ._llcrypt unregistered [ 1231.234253] Key type ._llcrypt registered [ 1231.237253] Key type .llcrypt registered [ 1231.317995] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 1248.155065] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1249.021195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1249.036450] alg: No test for adler32 (adler32-zlib) [ 1250.152963] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 1250.482644] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1252.183902] Key type lgssc registered [ 1253.176430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1286.389784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1286.411665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1287.722688] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1287.760246] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1287.833796] Lustre: lustre-MDT0000: new disk, initializing [ 1287.907876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1287.924799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1292.006269] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1301.595562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1301.687266] Lustre: 44061:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1301.725697] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1301.732378] Lustre: Skipped 1 previous similar message [ 1301.784092] Lustre: lustre-MDT0001: new disk, initializing [ 1301.863054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1301.894609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1301.902200] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1305.654970] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1312.635922] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1316.329716] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1323.419049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1323.691397] Lustre: lustre-OST0000: new disk, initializing [ 1323.698927] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1323.820091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1331.070967] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1332.814267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1332.823342] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1332.958270] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1340.189260] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1344.153559] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1344.283842] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1348.312162] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1348.599787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1353.706019] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1353.707093] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1353.720254] LustreError: Skipped 1 previous similar message [ 1353.749903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1353.756128] Lustre: Skipped 1 previous similar message [ 1358.820658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1358.826438] Lustre: Skipped 1 previous similar message [ 1363.941228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1363.958743] Lustre: Skipped 1 previous similar message [ 1364.959826] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1365.059614] Lustre: server umount lustre-OST0000 complete [ 1369.060077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1369.064103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1369.072050] Lustre: Skipped 2 previous similar messages [ 1369.090506] Lustre: Skipped 1 previous similar message [ 1374.185495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1374.196281] Lustre: Skipped 1 previous similar message [ 1374.896746] Lustre: server umount lustre-MDT0000 complete [ 1378.166767] LustreError: 44052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779991708 with bad export cookie 9510917284463000622 [ 1378.176846] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1378.184216] LustreError: 44052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1378.479517] Lustre: server umount lustre-MDT0001 complete [ 1387.230457] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 14:08:35 (1779991715) [ 1393.670658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1394.071219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1398.044776] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1399.265771] LustreError: 48114:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1399.281580] LustreError: 48114:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1403.362935] LustreError: 48114:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1405.632436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1405.937544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1409.517621] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1414.566722] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1418.092727] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1426.173797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1426.463168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1432.399361] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1438.342650] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1442.966910] LustreError: 50305:0:(llog.c:441:llog_init_handle()) llog type is not specified! [ 1443.950112] Lustre: Modifying parameter general.timeout=1 in log params [ 1445.736567] Lustre: Modifying parameter general.timeout=3 in log params [ 1445.739437] Lustre: Skipped 1 previous similar message [ 1448.353296] Lustre: Modifying parameter general.timeout=6 in log params [ 1448.356373] Lustre: Skipped 2 previous similar messages [ 1452.985391] Lustre: Modifying parameter general.timeout=11 in log params [ 1452.989153] Lustre: Skipped 4 previous similar messages [ 1461.147779] Lustre: Modifying parameter general.timeout=21 in log params [ 1461.156680] Lustre: Skipped 9 previous similar messages [ 1477.565558] Lustre: Modifying parameter general.timeout=40 in log params [ 1477.572698] Lustre: Skipped 18 previous similar messages [ 1509.922936] Lustre: Modifying parameter general.timeout=78 in log params [ 1509.930297] Lustre: Skipped 37 previous similar messages [ 1561.700306] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 14:11:30 (1779991890) [ 1564.404972] Lustre: 57240:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1566.071799] Lustre: 57349:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1569.334475] Lustre: 57546:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1569.348240] Lustre: 57546:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 1574.259974] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule= in log params [ 1574.272560] Lustre: Skipped 59 previous similar messages [ 1595.372674] Lustre: 59107:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1595.388764] Lustre: 59107:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 2 previous similar messages [ 1595.431351] LustreError: 59107:0:(mgs_handler.c:1166:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1599.623470] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 14:12:07 (1779991927) [ 1612.259905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1612.264937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1612.277719] Lustre: Skipped 2 previous similar messages [ 1612.280701] Lustre: Skipped 1 previous similar message [ 1617.610116] Lustre: server umount lustre-MDT0000 complete [ 1620.448214] LustreError: 48094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779991951 with bad export cookie 9510917284463001504 [ 1620.450621] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1620.462411] LustreError: 48094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1620.729527] Lustre: server umount lustre-MDT0001 complete [ 1634.505708] Lustre: server umount lustre-OST0000 complete [ 1659.494547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1659.890407] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1660.216639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1664.661405] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1673.715596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1673.801445] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1673.821730] Lustre: 61641:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1678.431684] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1684.426334] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1688.506929] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1694.952131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1695.122754] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1695.341686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1695.349348] Lustre: Skipped 1 previous similar message [ 1700.454547] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1705.433876] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1706.965259] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1706.977559] Lustre: Skipped 15 previous similar messages [ 1707.033567] Lustre: 63866:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1718.762698] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1718.774276] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1718.823870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1718.833605] Lustre: Skipped 4 previous similar messages [ 1720.787671] Lustre: server umount lustre-OST0000 complete [ 1724.395878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1724.405543] Lustre: Skipped 2 previous similar messages [ 1730.432698] Lustre: server umount lustre-MDT0000 complete [ 1733.681422] LustreError: 61634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779992064 with bad export cookie 9510917284463004458 [ 1733.688346] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1733.693164] LustreError: 61634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1734.136512] Lustre: server umount lustre-MDT0001 complete [ 1744.523233] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1746.913576] Key type lgssc unregistered [ 1747.134641] LNet: 65061:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1747.140497] LNetError: 65061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1747.152705] LNet: Removed LNI 192.168.203.139@tcp [ 1747.905085] Key type .llcrypt unregistered [ 1747.909759] Key type ._llcrypt unregistered [ 1769.637155] Key type ._llcrypt registered [ 1769.638843] Key type .llcrypt registered [ 1769.723395] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1770.859135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1770.868208] alg: No test for adler32 (adler32-zlib) [ 1771.906966] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 1772.187642] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1773.943210] Key type lgssc registered [ 1774.763392] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1782.577241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1782.601314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1784.223435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1788.661665] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1793.511316] LustreError: 66456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1793.531719] LustreError: 66456:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1797.447463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1798.109722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1802.073720] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1807.062289] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1810.347841] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1816.173664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1816.735180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1821.677732] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1826.604603] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1838.948162] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1849.312112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1849.323200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1849.346279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1849.829517] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1849.848140] Lustre: Skipped 1 previous similar message [ 1849.854396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1849.865756] Lustre: Skipped 1 previous similar message [ 1852.662677] Lustre: server umount lustre-MDT0000 complete [ 1854.951230] LustreError: 66456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1854.974363] LustreError: 66456:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1855.779259] LustreError: 66436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779992186 with bad export cookie 15545827405803307576 [ 1855.787759] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1855.788031] LustreError: 66436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1856.067896] Lustre: server umount lustre-MDT0001 complete [ 1869.730529] Lustre: server umount lustre-OST0000 complete [ 1895.290965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1895.563663] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1895.582815] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1895.782805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1900.177990] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1908.326090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1908.425512] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1908.453953] Lustre: 71005:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1908.877316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1911.996179] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1917.167278] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1920.659513] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1926.982658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1927.214346] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1927.677356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1933.082885] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1939.665386] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1941.800873] Lustre: 73242:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1944.085092] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1953.765864] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1953.777058] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1953.792955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1955.976458] Lustre: server umount lustre-OST0000 complete [ 1959.392671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1959.415183] Lustre: Skipped 1 previous similar message [ 1959.431355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1959.451022] Lustre: Skipped 3 previous similar messages [ 1964.515662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1964.521738] Lustre: Skipped 1 previous similar message [ 1973.224558] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1973.397067] Lustre: server umount lustre-MDT0000 complete [ 1974.756115] LustreError: 71014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1974.778639] LustreError: 71014:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1976.555053] LustreError: 72647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779992307 with bad export cookie 15545827405803308409 [ 1976.556796] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1976.577771] LustreError: 72647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1976.888289] Lustre: server umount lustre-MDT0001 complete [ 1987.430645] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1990.079922] Key type lgssc unregistered [ 1990.386941] LNet: 74439:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1990.399895] LNetError: 74439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1990.414930] LNet: Removed LNI 192.168.203.139@tcp [ 1991.273899] Key type .llcrypt unregistered [ 1991.279440] Key type ._llcrypt unregistered [ 2012.934832] Key type ._llcrypt registered [ 2012.936862] Key type .llcrypt registered [ 2013.032417] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2013.842602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2013.868291] alg: No test for adler32 (adler32-zlib) [ 2014.817281] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 2015.030371] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 2016.703164] Key type lgssc registered [ 2017.930890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2024.970825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2024.990816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2026.469266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2030.729547] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2036.705295] LustreError: 75836:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2036.736711] LustreError: 75836:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2039.146164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2039.867227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2043.963869] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2049.641814] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2053.315818] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2059.790543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2060.379220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2066.067527] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2071.486505] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2077.879559] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2078.753321] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2080.293983] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2080.296758] Lustre: Skipped 1 previous similar message [ 2082.519088] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2082.523598] Lustre: Skipped 2 previous similar messages [ 2087.295771] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2087.299082] Lustre: Skipped 5 previous similar messages [ 2095.802658] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2095.808914] Lustre: Skipped 10 previous similar messages [ 2111.885733] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2111.889795] Lustre: Skipped 21 previous similar messages [ 2144.400156] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2144.409676] Lustre: Skipped 48 previous similar messages [ 2155.208616] Lustre: 83586:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2157.722893] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 14:21:26 (1779992486) [ 2159.116168] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2160.628916] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 14:21:29 (1779992489) [ 2167.270074] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2167.284071] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2167.296158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2169.318667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2169.330726] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2169.331321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2170.525531] Lustre: server umount lustre-OST0000 complete [ 2174.440054] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2174.444802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2174.450810] Lustre: Skipped 1 previous similar message [ 2174.464447] Lustre: Skipped 2 previous similar messages [ 2179.554496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2179.567692] Lustre: Skipped 1 previous similar message [ 2184.675050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2184.678898] Lustre: Skipped 1 previous similar message [ 2188.255343] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2188.530875] Lustre: server umount lustre-MDT0000 complete [ 2189.795450] LustreError: 75836:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2189.825723] LustreError: 75836:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2192.268326] LustreError: 75816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779992523 with bad export cookie 15871159649529600229 [ 2192.272524] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2192.280024] LustreError: 75816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2192.587521] Lustre: server umount lustre-MDT0001 complete [ 2204.129601] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 2206.641396] Key type lgssc unregistered [ 2206.979283] LNet: 84818:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2206.987043] LNetError: 84818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2208.042068] LNet: Removed LNI 192.168.203.139@tcp [ 2208.807319] Key type .llcrypt unregistered [ 2208.811212] Key type ._llcrypt unregistered [ 2212.732354] Key type ._llcrypt registered [ 2212.733900] Key type .llcrypt registered [ 2212.807267] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2216.107198] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules [ 2216.940159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2216.959316] alg: No test for adler32 (adler32-zlib) [ 2217.969590] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 2218.133698] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 2219.792065] Key type lgssc registered [ 2220.831599] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2221.819911] LustreError: 85986:0:(lu_object.c:1434:lu_context_key_register()) cfs_race id 60d sleeping [ 2236.179266] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2243.850926] LustreError: 85986:0:(lu_object.c:1434:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2243.958264] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2243.989604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2245.529772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2249.177617] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2255.844248] LustreError: 86993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2255.860896] LustreError: 86993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2256.854495] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 14:23:05 (1779992585) [ 2260.960754] LustreError: 86993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2261.246140] LustreError: 87024:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2261.255216] LustreError: 87024:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2261.261297] LustreError: 87024:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 2261.583687] Lustre: server umount lustre-MDT0000 complete [ 2272.715483] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 2275.008873] Key type lgssc unregistered [ 2275.232690] LNet: 88482:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2275.246190] LNetError: 88482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2275.264155] LNet: Removed LNI 192.168.203.139@tcp [ 2276.089778] Key type .llcrypt unregistered [ 2276.092311] Key type ._llcrypt unregistered [ 2290.947685] Key type ._llcrypt registered [ 2290.949557] Key type .llcrypt registered [ 2291.085634] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2292.194401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2292.221613] alg: No test for adler32 (adler32-zlib) [ 2293.208698] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 2293.388556] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 2295.039151] Key type lgssc registered [ 2295.958269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2302.667686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2302.683192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2304.196600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2308.233954] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2313.697342] LustreError: 89682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2313.713091] LustreError: 89682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2316.315311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2316.843622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2320.949431] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2326.468539] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2329.946950] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2336.507837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2337.042671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2342.419392] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2346.807563] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2389.692932] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 14:25:18 (1779992718) [ 2390.859604] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2392.375499] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 14:25:21 (1779992721) [ 2398.689406] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2398.696525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2398.716661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2399.713647] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2399.716773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2399.722755] Lustre: Skipped 1 previous similar message [ 2399.726994] Lustre: Skipped 1 previous similar message [ 2401.391459] Lustre: server umount lustre-MDT0000 complete [ 2404.733880] LustreError: 89661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779992735 with bad export cookie 10799346274155609770 [ 2404.748178] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2404.755527] LustreError: 89661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2404.834443] LustreError: 90695:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2404.838419] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2404.874712] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2405.110665] Lustre: server umount lustre-MDT0001 complete [ 2414.987381] Lustre: server umount lustre-OST0000 complete [ 2426.271632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2426.806628] LustreError: 93617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2426.999702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2431.506739] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2432.486461] LustreError: 93617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2436.579917] LustreError: 93616:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2440.832889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2441.621608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2445.365728] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2451.504355] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2456.804732] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2469.890193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2470.075300] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2470.082333] LustreError: 93604:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2470.097179] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2470.103519] LustreError: 95475:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -98 [ 2470.108565] LustreError: 95475:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 2470.112879] LustreError: 95475:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2470.182195] Lustre: server umount lustre-OST0000 complete [ 2470.186279] LustreError: 95475:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2477.563301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2477.704726] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2477.715897] LustreError: 93605:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2477.746601] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2477.757493] LustreError: 95942:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -98 [ 2477.765490] LustreError: 95942:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 2477.773313] LustreError: 95942:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2477.859490] Lustre: server umount lustre-OST0000 complete [ 2477.869597] LustreError: 95942:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2485.804371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2486.008374] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2486.014822] Lustre: Found index 0 for lustre-OST0000, updating log [ 2486.022820] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2486.089387] Lustre: lustre-OST0000: new disk, initializing [ 2486.098812] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2486.351432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2491.721411] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2495.272968] Lustre: Failing over lustre-OST0000 [ 2495.403122] Lustre: server umount lustre-OST0000 complete [ 2501.601852] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2501.604985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2501.615235] Lustre: Skipped 1 previous similar message [ 2506.720397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2506.728797] Lustre: Skipped 2 previous similar messages [ 2512.863337] Lustre: 88995:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779992827/real 1779992827] req@0000000026b32146 x1866457545794432/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779992843 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2512.903636] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2513.375991] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2513.540040] Lustre: server umount lustre-MDT0000 complete [ 2513.887275] Lustre: 88993:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779992828/real 1779992828] req@000000002c77cc97 x1866457545794688/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1779992844 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2517.107238] LustreError: 93597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779992847 with bad export cookie 10799346274155612514 [ 2517.112623] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2517.120809] LustreError: 93597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2517.341915] Lustre: server umount lustre-MDT0001 complete [ 2525.413031] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 14:27:33 (1779992853) [ 2540.949886] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2550.593071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2550.922239] LustreError: 98793:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2551.044393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2554.969504] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2562.406285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2566.392184] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2569.187786] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2575.553613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2576.317783] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2581.695036] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2587.557786] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2587.583027] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2587.583190] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2589.505237] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2589.602722] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2589.686332] Lustre: lustre-OST0001: new disk, initializing [ 2589.691614] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2589.913683] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2589.918733] Lustre: Skipped 2 previous similar messages [ 2595.287745] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2595.357921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2595.377236] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2595.456613] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2602.670581] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2607.698626] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2616.803095] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2616.804964] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2616.811397] Lustre: Skipped 2 previous similar messages [ 2616.824745] Lustre: Skipped 6 previous similar messages [ 2622.624471] Lustre: server umount lustre-MDT0001 complete [ 2623.509435] LustreError: 98789:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2623.525644] LustreError: 98789:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2631.626813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2631.707265] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2631.710777] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2631.758496] Lustre: 98782:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2633.590959] LustreError: 102638:0:(obd_mount.c:212:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2633.603049] LustreError: 102638:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2633.617617] LustreError: 98784:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2633.763122] LustreError: 102644:0:(obd_config.c:2136:class_config_llog_handler()) MGC192.168.203.139@tcp: cfg command failed: rc = -17 [ 2633.794180] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2633.794180] [ 2636.546853] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2637.811921] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2645.191229] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 14:29:33 (1779992973) [ 2660.945578] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2678.032990] Lustre: 104640:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2697.636775] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2701.540364] Lustre: 105773:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2789.344924] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2789.355128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2789.359394] Lustre: Skipped 4 previous similar messages [ 2795.730157] Lustre: server umount lustre-MDT0000 complete [ 2799.565699] LustreError: 99517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779993130 with bad export cookie 10799346274155613508 [ 2799.566858] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2799.574899] LustreError: 99517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2799.593946] LustreError: 101077:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2799.607835] LustreError: 101077:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2799.779797] Lustre: server umount lustre-MDT0001 complete [ 2807.290565] Lustre: server umount lustre-OST0000 complete [ 2812.200695] Lustre: server umount lustre-OST0001 complete [ 2819.615471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2828.369431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2835.601253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2845.104628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2863.233556] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2871.418320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2871.441224] Lustre: lustre-MDT0000: reset Object Index mappings [ 2871.863206] LustreError: 109706:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2872.036855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2872.042466] Lustre: Skipped 1 previous similar message [ 2875.346694] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2882.049156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2882.075381] Lustre: lustre-MDT0001: reset Object Index mappings [ 2886.314117] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2888.563473] Lustre: 110816:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2894.081299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2896.684537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 2899.051576] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2905.595652] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2907.844046] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2908.925104] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 2908.925821] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2910.829231] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2917.852134] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2921.394450] Lustre: 112666:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2980.388294] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 14:35:08 (1779993308) [ 3006.438723] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3006.454398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3006.462363] Lustre: Skipped 6 previous similar messages [ 3006.476108] Lustre: Skipped 12 previous similar messages [ 3012.581287] Lustre: server umount lustre-MDT0000 complete [ 3016.379587] LustreError: 113023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779993347 with bad export cookie 10799346274155660380 [ 3016.388960] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3016.393806] LustreError: 113023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3016.672806] LustreError: 109701:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3016.700603] LustreError: 109701:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 3022.971288] Lustre: server umount lustre-MDT0001 complete [ 3041.247150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3041.473040] Lustre: server umount lustre-OST0000 complete [ 3052.121693] Lustre: server umount lustre-OST0001 complete [ 3058.523990] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 3069.004207] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 3106.549955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3106.884861] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3106.924492] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3107.058467] Lustre: lustre-MDT0000: new disk, initializing [ 3107.187780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3107.192978] Lustre: Skipped 3 previous similar messages [ 3107.209100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3112.089740] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3118.989842] Lustre: server umount lustre-MDT0000 complete [ 3127.651444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3128.036704] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3128.055382] Lustre: 116890:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3128.069237] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3131.430396] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3135.983354] Lustre: server umount lustre-MDT0000 complete [ 3144.714844] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 14:37:53 (1779993473) [ 3146.663863] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3148.616199] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 14:37:56 (1779993476) [ 3165.765530] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 3176.908113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3181.342451] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3189.579711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3189.750581] Lustre: 118909:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3189.832082] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3189.835722] Lustre: Skipped 1 previous similar message [ 3189.924710] Lustre: lustre-MDT0001: new disk, initializing [ 3190.103267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3190.113311] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3194.292441] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3199.073486] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3204.896768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3205.276205] Lustre: lustre-OST0000: new disk, initializing [ 3205.282643] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3206.964501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3206.973270] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3207.071922] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3210.793266] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3224.159039] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3224.517520] Lustre: lustre-OST0001: new disk, initializing [ 3224.530792] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3226.655735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3226.683573] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3226.867118] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3233.496408] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3243.545383] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3246.955572] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3623.295062] Lustre: 127703:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3623.318861] Lustre: 127703:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3623.326462] Lustre: 127703:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3623.332770] Lustre: 127703:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3623.347184] Lustre: 127703:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3623.361640] Lustre: 127703:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3931.077723] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 14:50:58 (1779994258) [ 3946.985831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3946.989155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3947.007069] Lustre: Skipped 7 previous similar messages [ 3947.016892] Lustre: Skipped 11 previous similar messages [ 3957.216967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3957.232528] Lustre: Skipped 7 previous similar messages [ 3960.287120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3960.807983] Lustre: server umount lustre-MDT0000 complete [ 3962.341616] LustreError: 122107:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3962.357727] LustreError: 122107:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3964.484772] LustreError: 120082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779994295 with bad export cookie 10799346274155728980 [ 3964.494162] LustreError: 120082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3964.497759] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3967.459461] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3967.486922] Lustre: Skipped 1 previous similar message [ 3971.146851] Lustre: server umount lustre-MDT0001 complete [ 3980.912188] Lustre: server umount lustre-OST0000 complete [ 3990.604980] Lustre: server umount lustre-OST0001 complete [ 3996.914667] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 4007.936891] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 4042.789687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4043.116658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4043.150514] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4043.266591] Lustre: lustre-MDT0000: new disk, initializing [ 4043.367852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4043.374320] Lustre: Skipped 5 previous similar messages [ 4043.397263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4047.851915] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4057.539140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4057.682300] Lustre: 130734:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4057.740560] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4057.743476] Lustre: Skipped 1 previous similar message [ 4057.814465] Lustre: lustre-MDT0001: new disk, initializing [ 4057.992732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4058.006760] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4062.656667] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4070.453847] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4074.038686] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4080.722452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4080.957302] Lustre: lustre-OST0000: new disk, initializing [ 4080.960944] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4081.084406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4081.087449] Lustre: Skipped 1 previous similar message [ 4082.462531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4082.466179] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4082.581353] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4087.380598] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4095.787392] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4104.746262] Lustre: lustre-MDD0000: changelog on [ 4107.365443] Lustre: lustre-MDD0001: changelog on [ 5986.779711] Lustre: lustre-MDD0001: changelog off [ 5988.549124] Lustre: 134513:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5989.271511] Lustre: lustre-MDD0000: changelog off [ 5997.544380] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5997.554845] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5997.568565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5997.576011] Lustre: Skipped 4 previous similar messages [ 6001.607390] Lustre: server umount lustre-OST0000 complete [ 6007.790134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6007.820373] Lustre: Skipped 1 previous similar message [ 6007.823654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6007.838748] Lustre: Skipped 2 previous similar messages [ 6012.890935] Lustre: server umount lustre-MDT0000 complete [ 6012.899509] LustreError: 130745:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6012.937101] LustreError: 130745:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6017.524674] LustreError: 130724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779996348 with bad export cookie 10799346274156545775 [ 6017.536848] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6017.546880] LustreError: 130724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6017.978219] Lustre: server umount lustre-MDT0001 complete [ 6030.582462] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 6033.466142] Key type lgssc unregistered [ 6033.770275] LNet: 141493:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6033.789760] LNetError: 141493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6033.810874] LNet: Removed LNI 192.168.203.139@tcp [ 6034.672150] Key type .llcrypt unregistered [ 6034.675389] Key type ._llcrypt unregistered [ 6055.582864] Key type ._llcrypt registered [ 6055.585116] Key type .llcrypt registered [ 6055.693399] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 6073.015379] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 6074.536745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6074.566450] alg: No test for adler32 (adler32-zlib) [ 6075.748398] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 6076.112414] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 6077.839164] Key type lgssc registered [ 6079.804791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6115.771988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6115.822325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6117.135943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6117.169493] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6117.269114] Lustre: lustre-MDT0000: new disk, initializing [ 6117.382233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6117.402514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6123.718508] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6139.685684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6139.788360] Lustre: 144660:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6139.895800] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6139.901592] Lustre: Skipped 1 previous similar message [ 6140.011307] Lustre: lustre-MDT0001: new disk, initializing [ 6140.139917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6140.180483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6140.192234] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6144.952847] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6152.344693] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6155.702664] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6162.133062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6162.457680] Lustre: lustre-OST0000: new disk, initializing [ 6162.471926] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6162.603563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6168.157113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6168.170903] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6168.260535] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6168.424902] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6175.533495] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6180.590616] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_module obdecho/obdecho [ 6182.336562] LustreError: 147278:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6182.347175] LustreError: 147278:0:(obd_config.c:880:class_setup()) setup ec failed (-22) [ 6183.948811] LustreError: 147375:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 6194.153352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6194.161486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6194.166325] Lustre: Skipped 1 previous similar message [ 6199.267246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6199.271118] Lustre: Skipped 3 previous similar messages [ 6199.609884] Lustre: server umount lustre-MDT0000 complete [ 6203.420163] LustreError: 145440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779996534 with bad export cookie 15591959021930363499 [ 6203.422460] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6203.431515] LustreError: 145440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6203.953685] Lustre: server umount lustre-MDT0001 complete [ 6218.781517] Lustre: server umount lustre-OST0000 complete [ 6245.297233] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 6254.960812] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 6295.075779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6295.508465] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6295.535756] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6295.632842] Lustre: lustre-MDT0000: new disk, initializing [ 6295.789134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6295.832081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6301.359307] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6314.855636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6315.012408] Lustre: 151058:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6315.112710] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6315.121790] Lustre: Skipped 1 previous similar message [ 6315.315678] Lustre: lustre-MDT0001: new disk, initializing [ 6315.529865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6315.595434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6315.626744] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6321.104087] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6329.245660] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6332.592264] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6339.765237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6340.128444] Lustre: lustre-OST0000: new disk, initializing [ 6340.139420] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6340.282220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6345.822313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6345.840373] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6345.941618] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6347.767958] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6357.490759] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6363.925527] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6364.263492] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6369.175587] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6369.397839] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6376.927881] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6376.930352] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6376.937050] LustreError: Skipped 1 previous similar message [ 6376.939824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6376.939831] Lustre: Skipped 1 previous similar message [ 6376.975032] Lustre: Skipped 2 previous similar messages [ 6378.319882] Lustre: server umount lustre-OST0000 complete [ 6382.048558] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6382.052623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6382.069272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6382.085400] Lustre: Skipped 2 previous similar messages [ 6387.168344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6387.172564] Lustre: Skipped 1 previous similar message [ 6396.384244] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6396.386051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6396.395535] Lustre: Skipped 3 previous similar messages [ 6396.678243] Lustre: server umount lustre-MDT0000 complete [ 6401.660924] LustreError: 151050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779996732 with bad export cookie 15591959021930364458 [ 6401.662749] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6401.670791] LustreError: 151050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6402.082294] Lustre: server umount lustre-MDT0001 complete [ 6406.269416] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 15:32:14 (1779996734) [ 6408.825810] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6411.139891] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 15:32:19 (1779996739) [ 6418.849561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6419.608240] LustreError: 155068:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6419.638682] LustreError: 155068:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6419.723688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6425.060641] LustreError: 155068:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6425.276834] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6430.182683] LustreError: 155068:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6435.301973] LustreError: 155068:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6435.648824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6440.960219] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6447.573851] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6451.437698] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6458.072649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6458.376861] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6458.386052] Lustre: Skipped 1 previous similar message [ 6464.392610] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6470.329433] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6476.363090] LustreError: 157310:0:(ldlm_resource.c:1170:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x200000bd1:0x30:0x0].0x0 (000000008dedf646) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 6476.415490] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.39@tcp (stopping) [ 6476.769084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6476.786428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6476.805475] Lustre: Skipped 1 previous similar message [ 6482.835667] Lustre: server umount lustre-MDT0000 complete [ 6484.961662] LustreError: 155067:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6484.982970] LustreError: 155067:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6486.529299] LustreError: 155049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779996817 with bad export cookie 15591959021930365340 [ 6486.539501] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6486.544922] LustreError: 155049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6490.083208] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6490.099328] Lustre: Skipped 2 previous similar messages [ 6492.989089] Lustre: server umount lustre-MDT0001 complete [ 6499.569198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6512.971133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6513.277088] LustreError: 158576:0:(llog.c:1643:llog_backup()) MGC192.168.203.139@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6513.286413] Lustre: 158576:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.203.139@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6516.296189] LustreError: 158583:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6516.315418] LustreError: 158583:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6532.612462] LustreError: 158581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6532.632695] LustreError: 158581:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 16 previous similar messages [ 6568.425936] LustreError: 156675:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6568.460431] LustreError: 156675:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 26 previous similar messages [ 6574.048457] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6574.057193] Lustre: 158576:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.203.139@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6600.159586] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6600.174320] Lustre: 155059:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.203.139@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6615.050363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6620.147849] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6620.417731] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6620.639270] LustreError: 155059:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 6630.647793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6632.431586] LustreError: 158581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6632.467799] LustreError: 158581:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 44 previous similar messages [ 6637.782678] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6638.061695] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6643.211666] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6645.206308] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6648.771933] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6650.494893] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6673.889711] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6673.904500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6673.910218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6673.910226] Lustre: Skipped 9 previous similar messages [ 6673.956744] Lustre: Skipped 2 previous similar messages [ 6674.457446] Lustre: server umount lustre-MDT0000 complete [ 6678.531674] LustreError: 157515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779997009 with bad export cookie 15591959021930371815 [ 6678.549175] LustreError: 157515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6678.549754] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6679.052518] Lustre: server umount lustre-MDT0001 complete [ 6693.189592] Lustre: server umount lustre-OST0000 complete [ 6713.348364] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 6723.238399] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 6760.771161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6761.033888] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6761.080808] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6761.198368] Lustre: lustre-MDT0000: new disk, initializing [ 6761.326276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6761.342584] Lustre: Skipped 1 previous similar message [ 6761.369247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6766.254304] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6776.821769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6776.918114] Lustre: 163425:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6777.278422] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6781.734486] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6791.274361] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6795.812814] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6804.009597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6804.296422] Lustre: lustre-OST0000: new disk, initializing [ 6804.299711] Lustre: Skipped 1 previous similar message [ 6804.308415] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6804.312502] Lustre: Skipped 2 previous similar messages [ 6809.680213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6809.706666] Lustre: Skipped 1 previous similar message [ 6809.722568] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6809.921120] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6810.674615] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6819.270622] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6824.154816] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6824.368860] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6828.734873] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6828.932981] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6835.680442] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6835.683030] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6835.697221] LustreError: Skipped 1 previous similar message [ 6835.698047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6835.733175] Lustre: Skipped 1 previous similar message [ 6835.738587] Lustre: Skipped 2 previous similar messages [ 6836.934988] Lustre: server umount lustre-OST0000 complete [ 6846.726148] Lustre: server umount lustre-MDT0000 complete [ 6850.245808] LustreError: 163417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779997181 with bad export cookie 15591959021930388650 [ 6850.248623] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6850.257108] LustreError: 163417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6850.687859] Lustre: server umount lustre-MDT0001 complete [ 6859.195411] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 15:39:47 (1779997187) [ 6865.856713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6866.338708] LustreError: 167431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6870.813520] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6878.947477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6883.446374] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6887.926558] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6891.621818] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6897.477162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6903.913824] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6909.415701] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6924.382850] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 15:40:52 (1779997252) [ 6930.913293] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6930.923564] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6930.939038] Lustre: Skipped 2 previous similar messages [ 6935.248239] Lustre: server umount lustre-OST0000 complete [ 6947.602311] Lustre: server umount lustre-MDT0000 complete [ 6951.054854] LustreError: 167411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779997281 with bad export cookie 15591959021930389532 [ 6951.059511] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6951.066649] LustreError: 167411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6951.498819] Lustre: server umount lustre-MDT0001 complete [ 6960.953923] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 6963.729433] Key type lgssc unregistered [ 6964.056663] LNet: 170804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6964.069523] LNetError: 170804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6964.099793] LNet: Removed LNI 192.168.203.139@tcp [ 6964.958760] Key type .llcrypt unregistered [ 6964.961708] Key type ._llcrypt unregistered [ 6983.988787] Key type ._llcrypt registered [ 6983.991568] Key type .llcrypt registered [ 6984.152577] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 6984.982153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6985.108331] alg: No test for adler32 (adler32-zlib) [ 6986.158850] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 6986.419816] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 6988.127321] Key type lgssc registered [ 6989.248207] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6997.033120] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6997.078498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7012.642230] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7017.759323] LustreError: 172034:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.139@tcp: failed processing log, type 4: rc = -110 [ 7022.943983] LustreError: 172034:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7022.961300] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7022.973295] LustreError: MGC192.168.203.139@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7022.993239] LustreError: 172034:0:(tgt_mount.c:1822:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7023.008834] LustreError: 172034:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -22 [ 7023.022289] LustreError: 172034:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 7023.112732] Lustre: server umount lustre-OST0000 complete [ 7023.116356] LustreError: 172034:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7031.122635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7032.593271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7037.051454] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7043.041230] LustreError: 172534:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7043.067406] LustreError: 172534:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7044.872537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7045.295116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7049.409489] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7055.932946] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7060.367765] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7067.319733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7067.453490] LustreError: 174122:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7067.461642] Lustre: 174122:0:(mgc_request_server.c:743:mgc_llog_local_copy()) MGC192.168.203.139@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7067.617133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7074.157416] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7080.377885] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7082.977100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7082.982995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7083.001233] Lustre: Skipped 2 previous similar messages [ 7083.014782] Lustre: Skipped 2 previous similar messages [ 7088.097911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7088.107728] Lustre: Skipped 2 previous similar messages [ 7088.860854] Lustre: server umount lustre-MDT0000 complete [ 7092.646023] LustreError: 174775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779997423 with bad export cookie 6180208573981493327 [ 7092.659448] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7093.040610] Lustre: server umount lustre-MDT0001 complete [ 7107.388949] Lustre: server umount lustre-OST0000 complete [ 7116.965336] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 7127.718220] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7162.971338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7163.219260] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7163.259853] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7163.368219] Lustre: lustre-MDT0000: new disk, initializing [ 7163.447544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7163.477658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7168.311498] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7179.312972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7179.435209] Lustre: 177503:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7179.532823] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7179.543291] Lustre: Skipped 1 previous similar message [ 7179.641884] Lustre: lustre-MDT0001: new disk, initializing [ 7179.787396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7179.843976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7179.858057] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7184.744569] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7192.255643] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7196.388964] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7203.238669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7203.494348] Lustre: lustre-OST0000: new disk, initializing [ 7203.498817] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7203.558579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7210.110621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7210.124292] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7210.254593] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7210.338586] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7217.917230] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7222.478341] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7222.726400] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7227.246962] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7227.570164] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7230.953573] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7230.954963] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7230.983772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7231.004908] Lustre: Skipped 1 previous similar message [ 7235.585080] Lustre: server umount lustre-OST0000 complete [ 7241.184584] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7241.186625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7241.202394] Lustre: Skipped 2 previous similar messages [ 7241.218729] Lustre: Skipped 1 previous similar message [ 7245.543831] Lustre: server umount lustre-MDT0000 complete [ 7246.307672] LustreError: 177516:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7246.325739] LustreError: 177516:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7249.176091] LustreError: 177495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779997579 with bad export cookie 6180208573981494048 [ 7249.179410] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7249.184746] LustreError: 177495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7249.581345] Lustre: server umount lustre-MDT0001 complete [ 7257.712850] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 15:46:26 (1779997586) [ 7276.229351] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7279.380793] Key type lgssc unregistered [ 7279.706176] LNet: 182024:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7279.710062] LNetError: 182024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7279.741972] LNet: Removed LNI 192.168.203.139@tcp [ 7280.500916] Key type .llcrypt unregistered [ 7280.504165] Key type ._llcrypt unregistered [ 7298.551323] Key type ._llcrypt registered [ 7298.555653] Key type .llcrypt registered [ 7298.685492] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7300.085130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7300.189629] alg: No test for adler32 (adler32-zlib) [ 7301.283625] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 7301.612467] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 7303.287160] Key type lgssc registered [ 7304.434102] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7311.608989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7311.622407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7313.037934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7317.048481] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7322.594823] LustreError: 183227:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7322.609470] LustreError: 183227:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7324.909235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7325.152696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7329.005340] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7334.411511] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7339.364698] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7348.178544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7348.524520] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7348.538165] LustreError: 184865:0:(llog.c:1684:llog_backup()) MGC192.168.203.139@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7348.563195] Lustre: 184865:0:(mgc_request_server.c:740:mgc_llog_local_copy()) MGC192.168.203.139@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7348.644442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7354.135239] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7359.837800] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7363.046291] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7363.047411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7363.062800] Lustre: Skipped 1 previous similar message [ 7363.085944] Lustre: Skipped 2 previous similar messages [ 7368.164548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7368.178669] Lustre: Skipped 2 previous similar messages [ 7373.281556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7373.290556] Lustre: Skipped 2 previous similar messages [ 7376.351123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7376.531529] Lustre: server umount lustre-MDT0000 complete [ 7378.405531] LustreError: 184249:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7378.427370] LustreError: 184249:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7380.142323] LustreError: 183207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779997710 with bad export cookie 16353364273811157520 [ 7380.163285] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7380.534264] Lustre: server umount lustre-MDT0001 complete [ 7393.640251] Lustre: server umount lustre-OST0000 complete [ 7402.283812] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 7412.946858] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7451.158779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7451.532741] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7451.565962] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7451.637324] Lustre: lustre-MDT0000: new disk, initializing [ 7451.703206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7451.715377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7456.035742] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7466.365563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7466.485797] Lustre: 188248:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7466.525296] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7466.527512] Lustre: Skipped 1 previous similar message [ 7466.531164] LustreError: 189014:0:(llog.c:1684:llog_backup()) MGC192.168.203.139@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7466.537244] LustreError: 189014:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 7466.541836] Lustre: 189014:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.203.139@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7466.557750] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7466.560991] Lustre: Skipped 1 previous similar message [ 7466.620966] Lustre: lustre-MDT0001: new disk, initializing [ 7466.718594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7466.767016] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7466.792729] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7470.646855] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7477.366742] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7481.042263] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7488.047112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7488.346489] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7488.348448] Lustre: Skipped 1 previous similar message [ 7488.351515] LustreError: 189965:0:(llog.c:1684:llog_backup()) MGC192.168.203.139@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7488.363782] LustreError: 189965:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 7488.369274] Lustre: 189965:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.203.139@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7488.374722] Lustre: 189965:0:(mgc_request_server.c:763:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7488.390553] Lustre: lustre-OST0000: new disk, initializing [ 7488.394618] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7488.504424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7494.736441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7494.751075] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7494.806235] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7494.873980] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7502.445406] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7507.009656] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7507.232804] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7511.867667] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7512.051460] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7515.627350] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7515.627499] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7515.632313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7515.638025] Lustre: Skipped 2 previous similar messages [ 7515.652262] Lustre: Skipped 1 previous similar message [ 7520.427323] Lustre: server umount lustre-OST0000 complete [ 7525.860490] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7525.866410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7525.879470] Lustre: Skipped 1 previous similar message [ 7530.379663] Lustre: server umount lustre-MDT0000 complete [ 7530.977712] LustreError: 188259:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7530.996416] LustreError: 188259:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7533.633608] LustreError: 188240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779997864 with bad export cookie 16353364273811158234 [ 7533.637164] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7533.650497] LustreError: 188240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7534.084274] Lustre: server umount lustre-MDT0001 complete [ 7542.492432] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 15:51:11 (1779997871) [ 7556.831307] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7565.975126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7566.399028] LustreError: 192951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7566.482372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7570.925946] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7571.951078] LustreError: 192951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7579.069495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7583.987970] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7586.882815] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7593.504064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7599.574398] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7607.872292] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7608.074746] Lustre: lustre-OST0001: new disk, initializing [ 7608.083695] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7608.211363] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7608.218518] Lustre: Skipped 2 previous similar messages [ 7613.542592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7613.548769] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7613.610494] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7613.743257] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7624.084104] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7634.703279] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7641.345524] Lustre: DEBUG MARKER: ADD OST3 [ 7646.465018] loop: module loaded [ 7646.534222] loop0: detected capacity change from 0 to 204800000 [ 7646.825811] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7646.852333] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7646.889471] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7652.929300] loop0: detected capacity change from 0 to 204800000 [ 7653.805317] Lustre: DEBUG MARKER: STOP OST3 [ 7659.331748] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7659.671433] Lustre: lustre-OST0003: new disk, initializing [ 7659.681341] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7661.662411] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7661.779382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7661.787632] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7661.838887] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7663.049872] Lustre: Failing over lustre-OST0003 [ 7663.272577] Lustre: server umount lustre-OST0003 complete [ 7663.717332] LustreError: 197384:0:(osp_precreate.c:1280:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7666.074111] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7666.722213] LustreError: 194403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7666.738822] LustreError: 194403:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7667.168156] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7667.173915] LustreError: Skipped 1 previous similar message [ 7667.176840] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7675.309729] Lustre: DEBUG MARKER: START OST3 again [ 7682.979796] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7683.471245] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7683.476515] Lustre: Skipped 1 previous similar message [ 7683.485424] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7684.775416] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7685.538296] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7685.538772] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7690.625250] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7710.279569] Lustre: Failing over lustre-OST0003 [ 7710.507454] Lustre: server umount lustre-OST0003 complete [ 7711.203302] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7711.203777] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7711.214626] LustreError: Skipped 1 previous similar message [ 7711.222586] Lustre: Skipped 1 previous similar message [ 7711.224135] LustreError: 194402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7711.250870] LustreError: 194402:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 7728.101411] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7728.109218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7728.124130] Lustre: Skipped 1 previous similar message [ 7728.130675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7728.134933] Lustre: Skipped 1 previous similar message [ 7734.182720] Lustre: server umount lustre-MDT0000 complete [ 7737.206050] LustreError: 192931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779998068 with bad export cookie 16353364273811159116 [ 7737.219894] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7737.227103] LustreError: 192931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7743.575490] Lustre: server umount lustre-MDT0001 complete [ 7747.580475] Lustre: server umount lustre-OST0000 complete [ 7751.515150] Lustre: server umount lustre-OST0001 complete [ 7757.809927] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 7765.856668] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7796.112466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7796.339542] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7796.387592] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7796.495400] Lustre: lustre-MDT0000: new disk, initializing [ 7796.613182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7799.863665] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7808.463794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7808.576310] Lustre: 202593:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7808.764964] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7812.161631] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7818.049572] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7821.159699] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7826.425893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7826.636327] Lustre: lustre-OST0000: new disk, initializing [ 7826.639051] Lustre: Skipped 1 previous similar message [ 7826.649348] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7826.658599] Lustre: Skipped 2 previous similar messages [ 7826.742786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7826.747548] Lustre: Skipped 2 previous similar messages [ 7828.623910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7828.645761] Lustre: Skipped 1 previous similar message [ 7828.652482] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7828.791716] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7832.378794] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7838.782760] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7842.700477] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7842.940458] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7847.135641] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7847.338919] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7849.442799] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7849.449934] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7849.458022] Lustre: Skipped 5 previous similar messages [ 7849.466390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7849.472793] Lustre: Skipped 9 previous similar messages [ 7863.263137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7863.393631] Lustre: server umount lustre-OST0000 complete [ 7872.922888] Lustre: server umount lustre-MDT0000 complete [ 7875.045560] LustreError: 203782:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7875.069824] LustreError: 203782:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 7876.206071] LustreError: 202586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779998207 with bad export cookie 16353364273811161419 [ 7876.210768] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7876.219799] LustreError: 202586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7876.641938] Lustre: server umount lustre-MDT0001 complete [ 7879.942873] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 15:56:48 (1779998208) [ 7894.218559] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 7903.928745] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7934.165033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7934.572923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7934.611189] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7934.787343] Lustre: lustre-MDT0000: new disk, initializing [ 7934.885565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7939.209297] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7950.938381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7951.100801] Lustre: 208763:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7951.439158] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7955.522863] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7962.417591] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7965.557416] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7971.580395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7973.335602] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7977.166921] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7984.027312] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7987.378317] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7987.581830] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7992.067917] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7992.311356] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7998.948302] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7998.955479] LustreError: Skipped 1 previous similar message [ 7998.963061] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7998.976236] Lustre: Skipped 3 previous similar messages [ 7998.980403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7998.984078] Lustre: Skipped 7 previous similar messages [ 8000.650247] Lustre: server umount lustre-OST0000 complete [ 8011.020429] Lustre: server umount lustre-MDT0000 complete [ 8014.313987] LustreError: 209912:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8014.331485] LustreError: 209912:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8014.375780] LustreError: 208755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779998345 with bad export cookie 16353364273811162301 [ 8014.377515] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8014.383027] LustreError: 208755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8014.686360] Lustre: server umount lustre-MDT0001 complete [ 8022.477971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8027.441607] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8035.592762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8039.970431] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8045.040812] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8048.276458] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8054.464150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8059.961888] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8064.620445] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8081.890995] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8081.901450] LustreError: Skipped 2 previous similar messages [ 8081.912363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8081.924219] Lustre: Skipped 3 previous similar messages [ 8081.933481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8081.940924] Lustre: Skipped 3 previous similar messages [ 8085.605243] Lustre: server umount lustre-OST0000 complete [ 8095.982963] Lustre: server umount lustre-MDT0000 complete [ 8100.019511] LustreError: 212557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779998430 with bad export cookie 16353364273811163183 [ 8100.035590] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8100.044115] LustreError: 212557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8100.378833] Lustre: server umount lustre-MDT0001 complete [ 8203.630746] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 8206.580849] Key type lgssc unregistered [ 8206.908213] LNet: 215919:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8206.929981] LNetError: 215919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8206.955260] LNet: Removed LNI 192.168.203.139@tcp [ 8207.726409] Key type .llcrypt unregistered [ 8207.733602] Key type ._llcrypt unregistered [ 8234.495362] Key type ._llcrypt registered [ 8234.497888] Key type .llcrypt registered [ 8234.672040] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 8250.664649] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 8251.426924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8251.478112] alg: No test for adler32 (adler32-zlib) [ 8252.458679] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 8252.664339] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 8254.343547] Key type lgssc registered [ 8255.401923] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8287.680203] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8287.706587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8289.002676] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8289.042838] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8289.116983] Lustre: lustre-MDT0000: new disk, initializing [ 8289.221646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8289.236624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8293.408234] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8304.125430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8304.248907] Lustre: 219238:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8304.299334] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8304.304454] Lustre: Skipped 1 previous similar message [ 8304.401655] Lustre: lustre-MDT0001: new disk, initializing [ 8304.513319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8304.575237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8304.590299] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8309.235364] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8317.440796] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8321.071780] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8327.443783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8327.666966] Lustre: lustre-OST0000: new disk, initializing [ 8327.674631] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8327.756074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8333.115874] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8335.934355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8335.946439] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8336.040592] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8339.588178] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8343.937641] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8344.119585] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8348.567723] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8348.822522] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8351.717725] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8351.719966] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8351.726861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8351.734019] Lustre: Skipped 1 previous similar message [ 8351.765859] LustreError: Skipped 1 previous similar message [ 8356.824368] Lustre: server umount lustre-OST0000 complete [ 8360.932806] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8360.945948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8360.960696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8360.967801] Lustre: Skipped 1 previous similar message [ 8361.954107] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8361.968327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8361.971380] Lustre: Skipped 1 previous similar message [ 8366.355969] Lustre: server umount lustre-MDT0000 complete [ 8369.469103] LustreError: 219229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779998700 with bad export cookie 15949148653787340105 [ 8369.473650] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8369.490756] LustreError: 219229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8369.791760] Lustre: server umount lustre-MDT0001 complete [ 8385.062565] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 8394.439297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8394.886086] LustreError: 223751:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8394.899651] LustreError: 223751:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8394.945507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8398.725259] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8400.352164] LustreError: 223751:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8404.451055] LustreError: 223751:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8406.531988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8406.897144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8411.208464] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8414.600604] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8420.328076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8420.661526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8428.222938] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8436.261771] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8436.423514] Lustre: lustre-OST0001: new disk, initializing [ 8436.427545] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8442.253840] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8444.560181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8444.568475] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8444.679367] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8452.919569] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8462.430709] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8479.957386] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 16:06:48 (1779998808) [ 8483.808478] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8483.823586] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8483.843078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8488.388953] Lustre: server umount lustre-MDT0000 complete [ 8490.985465] LustreError: 224461:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8491.006370] LustreError: 224461:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8492.213327] LustreError: 226009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779998823 with bad export cookie 15949148653787340987 [ 8492.214671] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8492.238187] LustreError: 226009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8492.619742] Lustre: server umount lustre-MDT0001 complete [ 8506.818832] Lustre: server umount lustre-OST0000 complete [ 8521.315477] Lustre: server umount lustre-OST0001 complete [ 8527.853549] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 8538.196347] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 8571.768064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8572.187569] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8572.243361] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8572.329534] Lustre: lustre-MDT0000: new disk, initializing [ 8572.426676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8572.436663] Lustre: Skipped 1 previous similar message [ 8572.472436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8576.923740] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8587.266039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8587.409242] Lustre: 229929:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8587.462214] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8587.466756] Lustre: Skipped 1 previous similar message [ 8587.556455] Lustre: lustre-MDT0001: new disk, initializing [ 8587.747445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8587.766556] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8592.304689] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8600.231937] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8604.049190] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8610.970994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8611.381772] Lustre: lustre-OST0000: new disk, initializing [ 8611.405778] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8611.551524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8611.555800] Lustre: Skipped 1 previous similar message [ 8613.370078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8613.385956] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8613.528128] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8617.757934] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8625.853406] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8630.457488] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8630.633035] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8634.769846] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8635.006105] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8638.944542] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8638.948683] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8638.957450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8638.966367] Lustre: Skipped 4 previous similar messages [ 8638.976503] Lustre: Skipped 5 previous similar messages [ 8649.191396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8649.205910] Lustre: Skipped 3 previous similar messages [ 8651.231294] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8651.474867] Lustre: server umount lustre-OST0000 complete [ 8659.433714] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8659.441942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8661.825607] Lustre: server umount lustre-MDT0000 complete [ 8664.546250] LustreError: 229940:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8664.582750] LustreError: 229940:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8665.675076] LustreError: 229921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779998996 with bad export cookie 15949148653787343129 [ 8665.698571] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8666.008992] Lustre: server umount lustre-MDT0001 complete [ 8674.541800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8675.012952] LustreError: 233739:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8679.140344] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8688.477424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8689.111484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8689.120972] Lustre: Skipped 1 previous similar message [ 8694.256500] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8700.715819] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8704.830988] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8711.547974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8718.817817] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8725.008852] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8872.932157] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8872.940450] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8872.944395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8872.951155] Lustre: Skipped 2 previous similar messages [ 8872.961136] Lustre: Skipped 3 previous similar messages [ 8885.215575] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8885.398050] Lustre: server umount lustre-OST0000 complete [ 8895.037262] Lustre: server umount lustre-MDT0000 complete [ 8898.532182] LustreError: 233738:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8898.558838] LustreError: 233738:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8898.601853] LustreError: 236234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779999229 with bad export cookie 15949148653787344011 [ 8898.611036] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8898.619324] LustreError: 236234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8899.038253] Lustre: server umount lustre-MDT0001 complete [ 8995.249844] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 8998.439971] Key type lgssc unregistered [ 8998.906438] LNet: 237001:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8998.920829] LNetError: 237001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8998.938798] LNet: Removed LNI 192.168.203.139@tcp [ 8999.758202] Key type .llcrypt unregistered [ 8999.760386] Key type ._llcrypt unregistered [ 9026.388265] Key type ._llcrypt registered [ 9026.392169] Key type .llcrypt registered [ 9026.555279] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 9043.200106] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 9044.345699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9044.390738] alg: No test for adler32 (adler32-zlib) [ 9045.580639] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 9045.871455] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 9047.567106] Key type lgssc registered [ 9048.824889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9096.952283] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 9107.263147] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9107.327486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9108.636311] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9108.673650] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9108.753306] Lustre: lustre-MDT0000: new disk, initializing [ 9108.812047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9108.828267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9113.039583] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9123.189883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9123.304379] Lustre: 241022:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9123.370180] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9123.374078] Lustre: Skipped 1 previous similar message [ 9123.448447] Lustre: lustre-MDT0001: new disk, initializing [ 9123.524194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9123.568160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9123.585231] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9127.878729] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9133.279348] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9140.515222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9140.902407] Lustre: lustre-OST0000: new disk, initializing [ 9140.911268] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9141.021834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9147.702252] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9151.022970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9151.034623] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9151.230487] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9158.410986] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9158.613441] Lustre: lustre-OST0001: new disk, initializing [ 9158.623839] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9158.775074] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9164.383389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9164.390266] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9164.476646] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9165.037928] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9176.550574] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9186.474301] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9204.509301] Lustre: server umount lustre-MDT0000 complete [ 9205.732774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9205.739933] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9207.757152] LustreError: 241798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779999538 with bad export cookie 11341792683305001356 [ 9207.760902] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9207.773104] LustreError: 241798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9208.330586] Lustre: server umount lustre-MDT0001 complete [ 9223.165887] Lustre: server umount lustre-OST0000 complete [ 9226.215719] Lustre: 238547:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779999541/real 1779999541] req@ffffa09249902680 x1866464625766528/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779999557 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9226.307925] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9226.319678] Lustre: Skipped 3 previous similar messages [ 9228.255177] Lustre: 238545:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779999543/real 1779999543] req@ffffa0935cf01500 x1866464625766784/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779999559 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9228.957251] Lustre: server umount lustre-OST0001 complete [ 9244.138996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9253.306743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9260.868731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9270.467966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9289.261408] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 9298.492192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9298.516257] Lustre: lustre-MDT0000: reset Object Index mappings [ 9299.020235] LustreError: 248448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9299.028080] LustreError: 248448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9299.088028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9302.561324] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9304.547702] LustreError: 248448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9308.644569] LustreError: 248448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9309.686299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9309.704042] Lustre: lustre-MDT0001: reset Object Index mappings [ 9310.025256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9313.431464] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9316.314902] Lustre: 249557:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9322.807658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9324.199219] LustreError: 249912:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9324.218512] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9329.264784] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9329.638290] LustreError: 249996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9337.870571] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9338.100557] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9338.109882] Lustre: Skipped 1 previous similar message [ 9339.558056] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9344.919601] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9352.396880] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9355.861312] Lustre: 251401:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9366.496821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9366.506582] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9366.538929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9367.797419] Lustre: server umount lustre-MDT0000 complete [ 9369.065177] LustreError: 249163:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9369.079331] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9369.084433] LustreError: 249163:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9369.100340] Lustre: Skipped 2 previous similar messages [ 9371.584435] LustreError: 248429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779999702 with bad export cookie 11341792683305008146 [ 9371.587917] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9371.599734] LustreError: 248429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9371.980975] Lustre: server umount lustre-MDT0001 complete [ 9386.054336] Lustre: server umount lustre-OST0000 complete [ 9400.196672] Lustre: server umount lustre-OST0001 complete [ 9419.664148] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 16:22:27 (1779999747) [ 9436.721877] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 9447.113460] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 9485.194527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9485.382994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9485.417705] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9485.492746] Lustre: lustre-MDT0000: new disk, initializing [ 9485.601839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9485.626497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9488.908372] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9498.614459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9498.732713] Lustre: 255621:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9498.785386] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9498.789594] Lustre: Skipped 1 previous similar message [ 9498.850364] Lustre: lustre-MDT0001: new disk, initializing [ 9498.958053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9498.976321] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9502.836666] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9511.239873] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9515.499433] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9522.611596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9522.968564] Lustre: lustre-OST0000: new disk, initializing [ 9522.971906] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9524.753527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9524.764400] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9524.900830] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9529.673720] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9537.081492] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9541.384464] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9541.531264] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9546.285409] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9546.508550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9550.305862] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9550.333910] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9550.353733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9554.782498] Lustre: server umount lustre-OST0000 complete [ 9560.550170] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9560.558867] LustreError: Skipped 1 previous similar message [ 9560.560935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9560.569116] Lustre: Skipped 1 previous similar message [ 9560.572179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9560.574909] Lustre: Skipped 1 previous similar message [ 9566.186369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9566.195825] Lustre: Skipped 3 previous similar messages [ 9570.281764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9570.291631] Lustre: Skipped 1 previous similar message [ 9572.319963] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9572.572197] Lustre: server umount lustre-MDT0000 complete [ 9575.394702] LustreError: 257256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9575.449835] LustreError: 257256:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9576.395827] LustreError: 255613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779999907 with bad export cookie 11341792683305010239 [ 9576.399299] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9576.403140] LustreError: 255613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9576.671273] Lustre: server umount lustre-MDT0001 complete [ 9595.779880] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 9605.293693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9605.689256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9605.694468] Lustre: Skipped 2 previous similar messages [ 9609.678893] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9610.738739] LustreError: 260131:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9610.770494] LustreError: 260131:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9617.085498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9621.631956] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9624.079914] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9629.251270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9634.959122] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9643.124992] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9643.271141] Lustre: lustre-OST0001: new disk, initializing [ 9643.274569] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9645.126603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9645.135163] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9645.220882] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9649.147422] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9657.706384] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9660.736636] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9670.003562] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9670.023516] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9676.263922] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9676.266071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9676.295040] Lustre: Skipped 4 previous similar messages [ 9676.301295] Lustre: Skipped 2 previous similar messages [ 9682.237885] Lustre: server umount lustre-MDT0000 complete [ 9685.398939] LustreError: 260112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780000016 with bad export cookie 11341792683305011121 [ 9685.409793] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9685.773849] Lustre: server umount lustre-MDT0001 complete [ 9698.466681] Lustre: server umount lustre-OST0000 complete [ 9712.405877] Lustre: server umount lustre-OST0001 complete [ 9727.394549] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 9736.202730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9736.633632] LustreError: 265415:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9736.653623] LustreError: 265415:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9736.718505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9736.724568] Lustre: Skipped 3 previous similar messages [ 9740.866762] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9749.243417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9753.825559] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9756.667553] Lustre: 266520:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9763.539263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9770.084135] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9777.982461] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9781.317519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9781.350784] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9783.452410] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9791.074400] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9795.021823] Lustre: 268371:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9804.798144] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9804.957851] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9813.940130] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 16:29:02 (1780000142) [ 9831.396067] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9831.411027] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9831.430217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9831.433075] Lustre: Skipped 5 previous similar messages [ 9834.641908] Lustre: server umount lustre-MDT0000 complete [ 9838.346078] LustreError: 268373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780000169 with bad export cookie 11341792683305017673 [ 9838.360147] LustreError: 268373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9838.366677] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9844.820941] Lustre: server umount lustre-MDT0001 complete [ 9854.532912] Lustre: server umount lustre-OST0000 complete [ 9858.243820] Lustre: server umount lustre-OST0001 complete [ 9864.491424] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 9874.790200] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 9924.664149] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 9935.574497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9935.842486] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9935.877830] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9935.981904] Lustre: lustre-MDT0000: new disk, initializing [ 9936.094358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9940.464306] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9950.287895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9950.483610] Lustre: 272561:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9950.856534] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9955.900068] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9967.879963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9970.016760] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9973.794526] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9987.952176] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9989.764710] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9989.771388] Lustre: Skipped 1 previous similar message [ 9989.895538] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9995.325722] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10007.012292] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10011.590908] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10011.597714] Lustre: Skipped 1 previous similar message [10030.391318] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 16:32:38 (1780000358) [10035.238630] Lustre: server umount lustre-MDT0000 complete [10036.206083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10036.222849] Lustre: Skipped 8 previous similar messages [10036.231768] LustreError: 272568:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10036.252384] LustreError: 272568:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [10038.810671] LustreError: 272552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780000369 with bad export cookie 11341792683305025324 [10038.811533] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10038.820485] LustreError: 272552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10039.155787] Lustre: server umount lustre-MDT0001 complete [10052.883081] Lustre: server umount lustre-OST0000 complete [10066.657864] Lustre: server umount lustre-OST0001 complete [10084.655974] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [10096.769808] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [10133.171444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10133.470612] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10133.510536] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10133.514121] Lustre: Skipped 4 previous similar messages [10133.587420] Lustre: lustre-MDT0000: new disk, initializing [10133.591922] Lustre: Skipped 3 previous similar messages [10133.713673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10133.725401] Lustre: Skipped 7 previous similar messages [10133.756415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10133.765750] Lustre: Skipped 3 previous similar messages [10138.299920] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10149.098708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10149.269355] Lustre: 279607:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10149.602273] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10154.231698] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10161.967320] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10165.096529] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10171.343490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10172.860161] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10179.509871] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10188.024316] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10190.012972] Lustre: Mounted lustre-client [10190.773547] LustreError: 282102:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffffa0924db44800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [10190.824871] Lustre: Unmounted lustre-client [10200.549656] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10200.558777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10200.565257] Lustre: Skipped 1 previous similar message [10200.572105] Lustre: Skipped 9 previous similar messages [10205.689244] Lustre: server umount lustre-OST0000 complete [10215.770796] Lustre: server umount lustre-MDT0000 complete [10219.369740] LustreError: 280386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780000550 with bad export cookie 11341792683305026850 [10219.370428] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10219.376749] LustreError: 280386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10219.900839] Lustre: server umount lustre-MDT0001 complete [10231.467273] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [10235.150717] Key type lgssc unregistered [10251.169485] LNet: 1 local NIs in recovery (showing 1): 192.168.203.139@tcp [10251.871639] LNet: 283384:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10251.875776] LNetError: 283384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10319.469667] LNet: Removed LNI 192.168.203.139@tcp [10320.270122] Key type .llcrypt unregistered [10320.272411] Key type ._llcrypt unregistered [10338.205808] Key type ._llcrypt registered [10338.208500] Key type .llcrypt registered [10338.306813] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [10357.281656] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [10359.017263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10359.172096] alg: No test for adler32 (adler32-zlib) [10360.322521] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [10360.574395] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [10362.335468] Key type lgssc registered [10364.229857] Lustre: Echo OBD driver; http://www.lustre.org/ [10396.257690] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 16:38:44 (1780000724) [10402.800491] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10402.856108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10404.165309] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10404.216093] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10404.309755] Lustre: lustre-MDT0000: new disk, initializing [10404.416525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10404.444432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10409.113244] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10421.991440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10422.226305] Lustre: 286566:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10422.294251] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10422.299328] Lustre: Skipped 1 previous similar message [10422.410473] Lustre: lustre-MDT0001: new disk, initializing [10422.555996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10422.601696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10422.620964] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10428.365184] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10438.008688] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10442.296271] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10449.389499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10449.776710] Lustre: lustre-OST0000: new disk, initializing [10449.783493] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10449.885120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10456.169107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10456.178593] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10456.302883] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10456.467815] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10464.357899] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10477.029821] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10477.034905] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10477.039127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10477.042850] Lustre: Skipped 1 previous similar message [10477.056668] Lustre: Skipped 1 previous similar message [10479.899379] Lustre: server umount lustre-OST0000 complete [10484.201340] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10484.212180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10484.224150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10487.277094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10487.281092] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10490.032524] Lustre: server umount lustre-MDT0000 complete [10493.940923] LustreError: 286558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780000824 with bad export cookie 3633176055411500180 [10493.944281] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10493.957470] LustreError: 286558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10494.379848] Lustre: server umount lustre-MDT0001 complete [10507.537754] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [10509.650733] Key type lgssc unregistered [10509.868695] LNet: 289943:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10509.874995] LNetError: 289943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10509.912513] LNet: Removed LNI 192.168.203.139@tcp [10510.892807] Key type .llcrypt unregistered [10510.895461] Key type ._llcrypt unregistered [10544.419539] Key type ._llcrypt registered [10544.429762] Key type .llcrypt registered [10544.571597] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [10545.883550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10545.911212] alg: No test for adler32 (adler32-zlib) [10546.866322] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [10547.080683] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [10548.751178] Key type lgssc registered [10549.589040] Lustre: Echo OBD driver; http://www.lustre.org/ [10557.013868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10557.039678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10558.594157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10563.315898] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10569.188114] LustreError: 291343:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10569.208755] LustreError: 291343:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10572.024077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10572.413234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10577.073116] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10582.977394] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10586.640360] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10593.481562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10594.096507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10600.215212] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10604.880354] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10614.162062] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10616.119796] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10623.015295] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.39@tcp (stopping) [10623.456171] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10623.460655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10624.993538] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10624.996360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10625.032820] Lustre: Skipped 3 previous similar messages [10626.822900] Lustre: server umount lustre-MDT0000 complete [10628.110791] LustreError: 291340:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10630.112828] LustreError: 291343:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10630.143124] LustreError: 291343:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10630.500446] LustreError: 291323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780000961 with bad export cookie 6048867187098209228 [10630.506541] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10630.510706] LustreError: 291323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10633.223222] LustreError: 291339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10633.245249] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.39@tcp (stopping) [10635.233289] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10635.252567] Lustre: Skipped 1 previous similar message [10635.257361] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10636.873776] Lustre: server umount lustre-MDT0001 complete [10640.359248] Lustre: server umount lustre-OST0000 complete [10648.016159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10652.491690] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10655.407880] Lustre: 295073:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.139@tcp [10657.630364] Lustre: server umount MGS complete [10665.663328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10666.167055] LustreError: 295648:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10666.198287] LustreError: 295648:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10666.239444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10670.974497] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10676.711974] LustreError: 295647:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10676.739867] LustreError: 295647:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10679.334424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10679.693715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10684.808764] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10693.949792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10694.319484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10700.777757] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10705.757854] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10707.396499] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [10711.804893] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10715.993671] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [10722.273750] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10722.283033] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10722.284497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10722.293398] Lustre: Skipped 1 previous similar message [10722.310252] Lustre: Skipped 1 previous similar message [10727.558987] Lustre: server umount lustre-OST0000 complete [10730.980911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10730.988248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10731.003182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10731.009715] Lustre: Skipped 2 previous similar messages [10745.311143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10745.503304] Lustre: server umount lustre-MDT0000 complete [10747.872573] LustreError: 295643:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10747.893828] LustreError: 295643:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10750.791912] LustreError: 295627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780001081 with bad export cookie 6048867187098210068 [10750.796683] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10750.818649] LustreError: 295627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10751.523149] Lustre: server umount lustre-MDT0001 complete [10766.061689] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [10769.531749] Key type lgssc unregistered [10769.725867] LNet: 298827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10769.741508] LNetError: 298827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10769.774327] LNet: Removed LNI 192.168.203.139@tcp [10770.614367] Key type .llcrypt unregistered [10770.616479] Key type ._llcrypt unregistered [10791.132420] Key type ._llcrypt registered [10791.136410] Key type .llcrypt registered [10791.359835] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [10807.534682] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [10808.382561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10808.517564] alg: No test for adler32 (adler32-zlib) [10809.597685] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [10809.849504] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [10811.527187] Key type lgssc registered [10812.737531] Lustre: Echo OBD driver; http://www.lustre.org/ [10851.157782] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10851.182394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10852.444564] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10852.468341] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10852.539166] Lustre: lustre-MDT0000: new disk, initializing [10852.618742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10852.639300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10857.093418] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10868.490854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10868.634305] Lustre: 301956:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10868.702156] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10868.710165] Lustre: Skipped 1 previous similar message [10868.844503] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10868.846058] Lustre: lustre-MDT0001: new disk, initializing [10868.938338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10868.974849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10868.994513] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10873.303588] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10881.934322] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10886.724096] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10894.124871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10894.333668] Lustre: lustre-OST0000: new disk, initializing [10894.344255] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10894.441623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10900.016591] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10902.592196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10902.610885] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10902.753183] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10907.061970] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10911.485413] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10911.661381] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10916.246596] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10916.558533] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10923.492730] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10923.500525] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10923.508353] LustreError: Skipped 1 previous similar message [10923.509232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10923.522121] Lustre: Skipped 1 previous similar message [10924.868899] Lustre: server umount lustre-OST0000 complete [10930.656435] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10930.670768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10930.686700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10930.689770] Lustre: Skipped 1 previous similar message [10933.733711] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10933.735757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10933.756963] Lustre: Skipped 1 previous similar message [10935.344460] LustreError: 305091:0:(ldlm_resource.c:1170:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (0000000071e10483) refcount nonzero (1) after lock cleanup; forcing cleanup. [10935.508676] Lustre: server umount lustre-MDT0000 complete [10939.764706] LustreError: 301949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780001270 with bad export cookie 423218091207947412 [10939.765957] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10939.785128] LustreError: 301949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10940.312691] Lustre: server umount lustre-MDT0001 complete [10949.644535] Lustre: DEBUG MARKER: == conf-sanity test 200a: check CPU partitions =========== 16:47:58 (1780001278) [11005.529906] Lustre: DEBUG MARKER: == conf-sanity test 200b: set CPU pattern using core selection ========================================================== 16:48:52 (1780001332) [11046.168095] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11048.845844] Key type lgssc unregistered [11049.223492] LNet: 308550:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11049.243590] LNetError: 308550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11049.274234] LNet: Removed LNI 192.168.203.139@tcp [11050.184821] Key type .llcrypt unregistered [11050.186546] Key type ._llcrypt unregistered [11129.397686] Key type ._llcrypt registered [11129.402658] Key type .llcrypt registered [11129.592192] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11129.952644] Key type .llcrypt unregistered [11129.955154] Key type ._llcrypt unregistered [11164.654583] Key type ._llcrypt registered [11164.657233] Key type .llcrypt registered [11164.807614] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11165.119410] Key type .llcrypt unregistered [11165.123033] Key type ._llcrypt unregistered [11197.381931] Key type ._llcrypt registered [11197.383953] Key type .llcrypt registered [11197.546581] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11197.842384] Key type .llcrypt unregistered [11197.846741] Key type ._llcrypt unregistered [11280.448358] Key type ._llcrypt registered [11280.454253] Key type .llcrypt registered [11280.588104] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11280.925272] Key type .llcrypt unregistered [11280.929313] Key type ._llcrypt unregistered [11315.445599] Key type ._llcrypt registered [11315.449095] Key type .llcrypt registered [11315.565936] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11315.968286] Key type .llcrypt unregistered [11315.974127] Key type ._llcrypt unregistered [11350.089934] Key type ._llcrypt registered [11350.091636] Key type .llcrypt registered [11350.165815] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11350.417516] Key type .llcrypt unregistered [11350.422572] Key type ._llcrypt unregistered [11383.820379] Key type ._llcrypt registered [11383.824432] Key type .llcrypt registered [11383.993921] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11384.354579] Key type .llcrypt unregistered [11384.359295] Key type ._llcrypt unregistered [11419.114887] Key type ._llcrypt registered [11419.120201] Key type .llcrypt registered [11419.257212] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11419.497716] Key type .llcrypt unregistered [11419.501217] Key type ._llcrypt unregistered [11452.515371] Key type ._llcrypt registered [11452.517245] Key type .llcrypt registered [11452.588700] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11452.891505] Key type .llcrypt unregistered [11452.896082] Key type ._llcrypt unregistered [11514.835202] Key type ._llcrypt registered [11514.836393] Key type .llcrypt registered [11515.014381] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11515.455602] Key type .llcrypt unregistered [11515.460403] Key type ._llcrypt unregistered [11546.348801] Key type ._llcrypt registered [11546.351787] Key type .llcrypt registered [11546.453400] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11546.729292] Key type .llcrypt unregistered [11546.734077] Key type ._llcrypt unregistered [11582.676893] Key type ._llcrypt registered [11582.687827] Key type .llcrypt registered [11582.889213] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11583.547860] Key type .llcrypt unregistered [11583.560827] Key type ._llcrypt unregistered [11643.733461] Key type ._llcrypt registered [11643.735346] Key type .llcrypt registered [11643.836809] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [11660.362470] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [11661.345538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11661.399802] alg: No test for adler32 (adler32-zlib) [11662.532914] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [11662.852256] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [11664.551351] Key type lgssc registered [11666.612922] Lustre: Echo OBD driver; http://www.lustre.org/ [11702.098631] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11409 sec ========= 17:00:30 (1780002030) [11704.075185] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 17:00:32 (1780002032) === [11707.753763] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 17:00:36 (1780002036) === [11747.587638] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11750.562407] Key type lgssc unregistered [11751.038431] LNet: 327759:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11751.051415] LNetError: 327759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11751.069232] LNet: Removed LNI 192.168.203.139@tcp [11751.905155] Key type .llcrypt unregistered [11751.909125] Key type ._llcrypt unregistered