[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 432181205 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001008] APIC: Switch to symmetric I/O mode setup [ 0.002259] x2apic enabled [ 0.003003] Switched APIC routing to physical x2apic. [ 0.004008] kvm-guest: setup PV IPIs [ 0.006000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006029] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007006] pid_max: default: 32768 minimum: 301 [ 0.008086] LSM: Security Framework initializing [ 0.009024] Yama: becoming mindful. [ 0.010018] SELinux: Initializing. [ 0.011033] *** VALIDATE selinux *** [ 0.017212] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.021628] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.022114] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024026] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025055] *** VALIDATE tmpfs *** [ 0.027102] *** VALIDATE proc *** [ 0.028155] *** VALIDATE cgroup *** [ 0.028874] *** VALIDATE cgroup2 *** [ 0.029186] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.030093] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.031003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.032018] Spectre V2 : User space: Vulnerable [ 0.033005] Speculative Store Bypass: Vulnerable [ 0.036429] debug: unmapping init [mem 0xffffffff9aa59000-0xffffffff9aa60fff] [ 0.038140] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.039593] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.040010] ... version: 2 [ 0.040907] ... bit width: 48 [ 0.041005] ... generic registers: 4 [ 0.041760] ... value mask: 0000ffffffffffff [ 0.042005] ... max period: 00007fffffffffff [ 0.043005] ... fixed-purpose events: 3 [ 0.043974] ... event mask: 000000070000000f [ 0.044232] rcu: Hierarchical SRCU implementation. [ 0.046132] smp: Bringing up secondary CPUs ... [ 0.047256] x86: Booting SMP configuration: [ 0.048016] .... node #0, CPUs: #1 #2 #3 [ 0.050554] smp: Brought up 1 node, 4 CPUs [ 0.051973] smpboot: Max logical packages: 1 [ 0.052008] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.134700] node 0 deferred pages initialised in 81ms [ 0.137474] devtmpfs: initialized [ 0.138224] x86/mm: Memory block size: 128MB [ 0.140787] gcov: version magic: 0x41383552 [ 0.143319] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.145089] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.147217] pinctrl core: initialized pinctrl subsystem [ 0.148098] [ 0.148755] ************************************************************* [ 0.151028] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153006] ** ** [ 0.155011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.158008] ** ** [ 0.160007] ** This means that this kernel is built to expose internal ** [ 0.161005] ** IOMMU data structures, which may compromise security on ** [ 0.163008] ** your system. ** [ 0.165009] ** ** [ 0.167006] ** If you see this message and you are not debugging the ** [ 0.168006] ** kernel, report this immediately to your vendor! ** [ 0.170008] ** ** [ 0.171004] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.173009] ************************************************************* [ 0.176717] NET: Registered protocol family 16 [ 0.177304] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.180039] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.182042] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.186034] cpuidle: using governor menu [ 0.187859] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.193174] PCI: Using configuration type 1 for base access [ 0.195140] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.205119] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.207030] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.209063] cryptd: max_cpu_qlen set to 1000 [ 0.211355] ACPI: Added _OSI(Module Device) [ 0.213015] ACPI: Added _OSI(Processor Device) [ 0.214010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.216011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.219247] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.225753] ACPI: Interpreter enabled [ 0.227058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.228010] ACPI: Using IOAPIC for interrupt routing [ 0.231114] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.235772] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.243686] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.246030] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.247013] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.250086] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.254354] acpiphp: Slot [2] registered [ 0.255079] acpiphp: Slot [5] registered [ 0.256097] acpiphp: Slot [6] registered [ 0.258049] acpiphp: Slot [7] registered [ 0.259000] acpiphp: Slot [8] registered [ 0.260089] acpiphp: Slot [9] registered [ 0.261082] acpiphp: Slot [10] registered [ 0.262091] acpiphp: Slot [3] registered [ 0.263055] acpiphp: Slot [4] registered [ 0.264063] acpiphp: Slot [11] registered [ 0.265110] acpiphp: Slot [12] registered [ 0.267090] acpiphp: Slot [13] registered [ 0.268075] acpiphp: Slot [14] registered [ 0.269060] acpiphp: Slot [15] registered [ 0.270066] acpiphp: Slot [16] registered [ 0.271060] acpiphp: Slot [17] registered [ 0.273060] acpiphp: Slot [18] registered [ 0.274095] acpiphp: Slot [19] registered [ 0.275058] acpiphp: Slot [20] registered [ 0.277179] acpiphp: Slot [21] registered [ 0.279207] acpiphp: Slot [22] registered [ 0.282185] acpiphp: Slot [23] registered [ 0.285150] acpiphp: Slot [24] registered [ 0.286076] acpiphp: Slot [25] registered [ 0.288096] acpiphp: Slot [26] registered [ 0.289079] acpiphp: Slot [27] registered [ 0.290067] acpiphp: Slot [28] registered [ 0.293145] acpiphp: Slot [29] registered [ 0.295071] acpiphp: Slot [30] registered [ 0.296059] acpiphp: Slot [31] registered [ 0.297069] PCI host bridge to bus 0000:00 [ 0.298014] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.301034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.304009] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.306013] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.309014] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.312013] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.314200] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.318051] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.321294] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.329012] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.333781] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.336016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.338009] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.341016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.344112] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.347664] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.349020] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.350862] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.354762] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.366016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.370007] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.376297] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.390023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.398010] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.410019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.419734] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.424027] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.429016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.443013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.456489] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.463011] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.470035] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.490015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.499000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.506012] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.512012] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.526015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.533930] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.539009] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.546012] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.565022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.577027] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.584012] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.589012] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.610013] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.628324] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.630826] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.635720] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.638370] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.640212] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.646132] iommu: Default domain type: Passthrough [ 0.649720] SCSI subsystem initialized [ 0.653597] ACPI: bus type USB registered [ 0.656067] usbcore: registered new interface driver usbfs [ 0.659079] usbcore: registered new interface driver hub [ 0.661139] usbcore: registered new device driver usb [ 0.662139] pps_core: LinuxPPS API ver. 1 registered [ 0.664007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.667065] PTP clock support registered [ 0.668129] EDAC MC: Ver: 3.0.0 [ 0.671177] PCI: Using ACPI for IRQ routing [ 0.673737] NetLabel: Initializing [ 0.676009] NetLabel: domain hash size = 128 [ 0.677008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.681091] NetLabel: unlabeled traffic allowed by default [ 0.685037] vgaarb: loaded [ 0.687011] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.689009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.694305] clocksource: Switched to clocksource kvm-clock [ 0.804436] VFS: Disk quotas dquot_6.6.0 [ 0.806193] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.809311] *** VALIDATE ramfs *** [ 0.811160] *** VALIDATE hugetlbfs *** [ 0.813474] pnp: PnP ACPI init [ 0.816875] pnp: PnP ACPI: found 6 devices [ 0.837411] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.840651] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.843235] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.845276] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.847240] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.849620] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.852372] NET: Registered protocol family 2 [ 0.854900] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.859729] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.864184] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.875919] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.879428] TCP: Hash tables configured (established 65536 bind 65536) [ 0.882432] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.885424] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.888115] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.891361] NET: Registered protocol family 1 [ 0.893943] RPC: Registered named UNIX socket transport module. [ 0.899911] RPC: Registered udp transport module. [ 0.904726] RPC: Registered tcp transport module. [ 0.908989] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.914338] NET: Registered protocol family 44 [ 0.916983] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.919742] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.923387] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.926866] PCI: CLS 0 bytes, default 64 [ 0.931279] Unpacking initramfs... [ 2.333473] debug: unmapping init [mem 0xffff9e6bbcc54000-0xffff9e6bbffbffff] [ 2.342362] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.347524] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.350061] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.978879] Initialise system trusted keyrings [ 2.980529] Key type blacklist registered [ 2.982087] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.990941] zbud: loaded [ 2.993711] *** VALIDATE nfs *** [ 2.994732] *** VALIDATE nfs4 *** [ 2.996222] pstore: using deflate compression [ 2.999768] Platform Keyring initialized [ 3.170873] NET: Registered protocol family 38 [ 3.172352] Key type asymmetric registered [ 3.173926] Asymmetric key parser 'x509' registered [ 3.177818] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.181567] io scheduler mq-deadline registered [ 3.183350] io scheduler kyber registered [ 3.187624] io scheduler bfq registered [ 3.191547] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.195984] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.200876] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.205457] ACPI: Power Button [PWRF] [ 3.216778] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.234502] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.256937] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.272532] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.309687] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.357308] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.402637] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.406939] Non-volatile memory driver v1.3 [ 3.408349] Linux agpgart interface v0.103 [ 3.449735] virtio_blk virtio1: [vda] 149768 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.452366] vda: detected capacity change from 0 to 76681216 [ 3.479753] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.482838] vdb: detected capacity change from 0 to 1073741824 [ 3.523936] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.530179] vdc: detected capacity change from 0 to 2621440000 [ 3.564664] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.569549] vdd: detected capacity change from 0 to 2621440000 [ 3.607267] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.610090] vde: detected capacity change from 0 to 4294967296 [ 3.636565] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.640110] vdf: detected capacity change from 0 to 4294967296 [ 3.647153] libphy: Fixed MDIO Bus: probed [ 3.653435] usbcore: registered new interface driver usbserial_generic [ 3.656882] usbserial: USB Serial support registered for generic [ 3.660317] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.666136] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.668837] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.671836] mousedev: PS/2 mouse device common for all mice [ 3.676789] rtc_cmos 00:05: RTC can wake from S4 [ 3.680115] rtc_cmos 00:05: registered as rtc0 [ 3.682977] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.686558] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.689588] intel_pstate: CPU model not supported [ 3.698329] hid: raw HID events driver (C) Jiri Kosina [ 3.701134] usbcore: registered new interface driver usbhid [ 3.703513] usbhid: USB HID core driver [ 3.705453] drop_monitor: Initializing network drop monitor service [ 3.708225] Initializing XFRM netlink socket [ 3.711048] NET: Registered protocol family 10 [ 3.713965] Segment Routing with IPv6 [ 3.715804] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.716116] NET: Registered protocol family 17 [ 3.716414] mpls_gso: MPLS GSO support [ 3.724436] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.732977] RAS: Correctable Errors collector initialized. [ 3.735321] AVX version of gcm_enc/dec engaged. [ 3.737303] AES CTR mode by8 optimization enabled [ 3.808606] sched_clock: Marking stable (3808535756, 0)->(4553915778, -745380022) [ 3.812682] registered taskstats version 1 [ 3.814912] Loading compiled-in X.509 certificates [ 3.819278] zswap: loaded using pool lzo/zbud [ 3.844447] Key type big_key registered [ 3.857685] Key type encrypted registered [ 3.859739] ima: No TPM chip found, activating TPM-bypass! [ 3.862348] ima: Allocated hash algorithm: sha1 [ 3.864645] ima: No architecture policies found [ 3.866866] evm: Initialising EVM extended attributes: [ 3.869251] evm: security.selinux [ 3.870791] evm: security.ima [ 3.872295] evm: security.capability [ 3.873881] evm: HMAC attrs: 0x1 [ 3.876736] rtc_cmos 00:05: setting system clock to 2026-09-06 23:36:41 UTC (1788737801) [ 3.883859] debug: unmapping init [mem 0xffffffff9ba03000-0xffffffff9bbfffff] [ 3.887642] debug: unmapping init [mem 0xffffffff9a782000-0xffffffff9aa58fff] [ 3.903304] Write protecting the kernel read-only data: 28672k [ 3.912922] debug: unmapping init [mem 0xffffffff98e03000-0xffffffff98ffffff] [ 3.919812] debug: unmapping init [mem 0xffffffff99714000-0xffffffff997fffff] [ 3.971282] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.978804] systemd[1]: Detected virtualization kvm. [ 3.980581] systemd[1]: Detected architecture x86-64. [ 3.985791] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.051439] systemd[1]: No hostname configured. [ 4.054329] systemd[1]: Set hostname to . [ 4.056639] random: systemd: uninitialized urandom read (16 bytes read) [ 4.062248] systemd[1]: Initializing machine ID from random generator. [ 4.139179] random: ln: uninitialized urandom read (6 bytes read) [ 4.303827] random: systemd: uninitialized urandom read (16 bytes read) [ 4.306178] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.317778] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.325079] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.413285] device-mapper: uevent: version 1.0.3 [ 5.420148] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.699719] scsi host0: ata_piix [ 6.728706] virtio_net virtio0 ens2: renamed from eth0 [ 6.744336] random: fast init done [ 6.777625] scsi host1: ata_piix [ 6.779223] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.782050] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.390579] random: crng init done [ 11.392272] random: 7 urandom warning(s) missed due to ratelimiting [ 11.913622] dracut-initqueue[586]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 12.994422] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.979926] printk: systemd: 26 output lines suppressed due to ratelimiting [ 15.494867] SELinux: Disabled at runtime. [ 15.566377] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.583247] systemd[1]: Detected virtualization kvm. [ 15.586312] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.299440] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.305163] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.315197] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.328739] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.336821] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.348868] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.365174] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ 16.599891] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 17.000903] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 17.352430] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 17.457406] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 17.674243] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 17.701346] EDAC sbridge: Ver: 1.1.2 [ 20.941064] Key type dns_resolver registered [ 21.889403] NFS: Registering the id_resolver key type [ 21.895162] Key type id_resolver registered [ 21.896354] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (6s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg342-server login: [ 46.290523] libcfs: loading out-of-tree module taints kernel. [ 46.306408] Key type ._llcrypt registered [ 46.307413] Key type .llcrypt registered [ 46.344411] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 53.195253] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 53.663218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 53.669222] alg: No test for adler32 (adler32-zlib) [ 54.644607] Lustre: Lustre: Build Version: 2.17.58_3_g671df26 [ 54.944695] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 56.551124] Key type lgssc registered [ 57.053227] Lustre: Echo OBD driver; http://www.lustre.org/ [ 62.906952] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 75.243529] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 79.317834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 79.325401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 80.407527] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 80.417429] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 80.455698] Lustre: lustre-MDT0000: new disk, initializing [ 80.479259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 80.484933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 81.753839] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 86.626546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 86.654715] Lustre: 6478:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 86.664958] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 86.666543] Lustre: Skipped 1 previous similar message [ 86.700995] Lustre: lustre-MDT0001: new disk, initializing [ 86.717947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 86.725444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 86.728029] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 87.952890] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 90.132369] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 93.265915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 93.364084] Lustre: lustre-OST0000: new disk, initializing [ 93.365910] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 93.368505] Lustre: 8419:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 93.387733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 95.362820] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 100.763764] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 100.807367] Lustre: lustre-OST0001: new disk, initializing [ 100.809195] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 100.811525] Lustre: 9479:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 100.827377] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 101.150688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 101.154181] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 101.190911] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 101.190913] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 102.851450] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 108.578890] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 111.054125] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 112.806713] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing check_logdir /tmp/testlogs/ [ 114.267959] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing yml_node [ 115.476576] Lustre: DEBUG MARKER: Client: 2.17.58.3 [ 116.268236] Lustre: DEBUG MARKER: MDS: 2.17.58.3 [ 117.064651] Lustre: DEBUG MARKER: OSS: 2.17.58.3 [ 117.558870] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Sep 6 19:38:34 EDT 2026 [ 122.700688] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 123.178436] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 137.183949] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 137.184710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 137.187977] Lustre: Skipped 2 previous similar messages [ 137.189567] Lustre: Skipped 1 previous similar message [ 140.888322] Lustre: server umount lustre-MDT0000 complete [ 142.035858] LustreError: 6472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788737939 with bad export cookie 1516269221047693579 [ 142.037660] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 142.039581] LustreError: 6472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 142.303571] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 142.303882] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 142.307034] Lustre: Skipped 2 previous similar messages [ 142.309746] Lustre: Skipped 3 previous similar messages [ 147.423596] LustreError: 6485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 147.423619] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 147.428458] LustreError: 6485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 147.431448] Lustre: Skipped 1 previous similar message [ 148.189947] Lustre: server umount lustre-MDT0001 complete [ 155.449641] Lustre: server umount lustre-OST0000 complete [ 162.631086] Lustre: server umount lustre-OST0001 complete [ 164.582329] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 167.342709] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 178.521626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 178.611435] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 178.624305] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 178.662472] Lustre: lustre-MDT0000: new disk, initializing [ 178.689716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 178.697942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 178.701172] Lustre: Skipped 1 previous similar message [ 180.119397] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 184.162643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 184.214846] Lustre: 15028:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 184.318501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 184.322696] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 184.325052] Lustre: Skipped 1 previous similar message [ 185.609491] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 189.112323] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 190.744126] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 192.718346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 192.819969] Lustre: lustre-OST0000: new disk, initializing [ 192.821204] Lustre: Skipped 1 previous similar message [ 192.822459] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 192.824319] Lustre: Skipped 2 previous similar messages [ 192.826814] Lustre: 16999:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 192.857870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 192.861062] Lustre: Skipped 1 previous similar message [ 194.366111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 194.369272] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 194.415124] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 194.679151] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 198.175934] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 199.521062] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 199.593595] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 200.855205] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 200.917718] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 204.772085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 204.772615] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 204.776569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 204.777752] LustreError: Skipped 1 previous similar message [ 207.627348] Lustre: server umount lustre-OST0000 complete [ 209.888612] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 209.888785] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 209.889441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 209.889445] Lustre: Skipped 1 previous similar message [ 209.899358] Lustre: Skipped 2 previous similar messages [ 220.128134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 220.129846] Lustre: Skipped 4 previous similar messages [ 223.199078] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 223.258814] Lustre: server umount lustre-MDT0000 complete [ 224.417404] LustreError: 15022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788738022 with bad export cookie 1516269221047695000 [ 224.422471] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 224.530299] Lustre: server umount lustre-MDT0001 complete [ 226.393160] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 19:40:23 (1788738023) [ 232.641862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 232.798592] LustreError: 19867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 232.822584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 234.158991] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 237.039984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 238.561332] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 241.007250] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 242.830201] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 243.527934] Lustre: Failing over lustre-MDT0000 [ 243.612598] Lustre: server umount lustre-MDT0000 complete [ 246.624716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 246.625423] LustreError: 19892:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 246.629885] Lustre: Skipped 1 previous similar message [ 251.871984] LustreError: 19886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 251.877105] LustreError: 19886:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 256.571143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 256.608237] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 257.977203] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 261.655544] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 261.792316] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 261.794955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 261.801756] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 267.232783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 267.233071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 267.241202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 267.243262] Lustre: Skipped 1 previous similar message [ 269.648770] Lustre: server umount lustre-MDT0000 complete [ 270.861165] LustreError: 19850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788738068 with bad export cookie 1516269221047696414 [ 270.864037] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 270.865145] LustreError: 19850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 270.975105] Lustre: server umount lustre-MDT0001 complete [ 273.733563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 273.876650] LustreError: 23200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 273.897575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 273.900447] Lustre: Skipped 2 previous similar messages [ 275.175897] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 277.929041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 279.344024] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 281.838551] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 283.761891] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 284.457880] Lustre: Failing over lustre-MDT0000 [ 284.547571] Lustre: server umount lustre-MDT0000 complete [ 287.520527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 287.523129] LustreError: 23195:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 287.524357] Lustre: Skipped 2 previous similar messages [ 297.546888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 297.588265] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 299.004489] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 302.813832] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 302.819724] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 302.822939] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 302.825381] Lustre: Skipped 1 previous similar message [ 302.830947] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 304.784397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 304.786364] Lustre: Skipped 1 previous similar message [ 310.938136] Lustre: server umount lustre-MDT0000 complete [ 312.248305] LustreError: 23226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788738109 with bad export cookie 1516269221047697352 [ 312.251693] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 312.252453] LustreError: 23226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 312.362510] Lustre: server umount lustre-MDT0001 complete [ 315.334930] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 19:41:52 (1788738112) [ 320.539892] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 323.500285] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 336.243797] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 339.983906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 340.059797] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 340.070945] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 340.104898] Lustre: lustre-MDT0000: new disk, initializing [ 340.128738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 340.131269] Lustre: Skipped 2 previous similar messages [ 340.136383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 341.420413] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 345.355261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 345.395054] Lustre: 29563:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 345.493052] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 346.713946] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 350.335688] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 352.052418] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 354.142352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 354.252400] Lustre: 31531:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 355.452965] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 355.468612] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 355.491686] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 355.499294] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 356.240956] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 359.851103] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 361.839843] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 366.051336] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 366.052893] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 366.055596] Lustre: Skipped 2 previous similar messages [ 366.057789] LustreError: Skipped 1 previous similar message [ 371.169722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 371.172817] Lustre: Skipped 5 previous similar messages [ 371.226481] Lustre: server umount lustre-OST0000 complete [ 376.289066] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 378.580161] Lustre: server umount lustre-MDT0000 complete [ 379.766155] LustreError: 29554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788738177 with bad export cookie 1516269221047697758 [ 379.769409] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 379.769525] LustreError: 29554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 379.944284] Lustre: server umount lustre-MDT0001 complete [ 383.554348] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 384.530390] Key type lgssc unregistered [ 384.651812] LNet: 33480:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 384.655462] LNetError: 33480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 384.662295] LNet: Removed LNI 192.168.203.142@tcp [ 384.997115] Key type .llcrypt unregistered [ 384.998325] Key type ._llcrypt unregistered [ 393.968394] Key type ._llcrypt registered [ 393.969512] Key type .llcrypt registered [ 394.003232] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 394.338965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 394.347054] alg: No test for adler32 (adler32-zlib) [ 395.188713] Lustre: Lustre: Build Version: 2.17.58_3_g671df26 [ 395.263405] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 396.839150] Key type lgssc registered [ 397.148887] Lustre: Echo OBD driver; http://www.lustre.org/ [ 399.567321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 399.573687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 400.715787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 401.967596] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 404.653248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 404.773402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 406.082531] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 408.394476] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 410.221319] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 412.356165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 412.468959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 414.234231] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 414.502032] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 416.434313] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 431.447913] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 19:43:48 (1788738228) [ 432.020155] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 432.578602] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 434.652276] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 19:43:51 (1788738231) [ 437.293580] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 19:43:54 (1788738234) [ 437.611835] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 440.214410] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 440.216421] Lustre: Skipped 2 previous similar messages [ 440.771019] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 19:43:57 (1788738237) [ 447.381798] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 447.385016] Lustre: Skipped 10 previous similar messages [ 448.206673] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 19:44:05 (1788738245) [ 449.097919] Lustre: *** cfs_fail_loc=131b, val=2*** [ 450.175886] Lustre: *** cfs_fail_loc=131b, val=2*** [ 452.718783] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 19:44:09 (1788738249) [ 456.700271] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 456.702941] Lustre: Skipped 7 previous similar messages [ 457.211607] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 19:44:14 (1788738254) [ 460.767717] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 460.770599] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 460.774523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 461.792508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 461.793559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 461.793632] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 461.802022] Lustre: Skipped 1 previous similar message [ 466.912679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 466.915020] Lustre: Skipped 1 previous similar message [ 472.033067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 472.035547] Lustre: Skipped 1 previous similar message [ 475.103131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 475.151825] Lustre: server umount lustre-OST0000 complete [ 476.639703] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 476.641805] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 476.645803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 482.453057] Lustre: server umount lustre-MDT0000 complete [ 483.622172] LustreError: 34848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788738281 with bad export cookie 17554451153212820895 [ 483.625281] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 483.626073] LustreError: 34848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 483.751216] Lustre: server umount lustre-MDT0001 complete [ 487.249172] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 488.200465] Key type lgssc unregistered [ 488.324299] LNet: 42696:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 488.326627] LNetError: 42696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 488.338302] LNet: Removed LNI 192.168.203.142@tcp [ 488.618162] Key type .llcrypt unregistered [ 488.619712] Key type ._llcrypt unregistered [ 494.032179] Key type ._llcrypt registered [ 494.033450] Key type .llcrypt registered [ 494.067954] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 500.179962] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 500.616220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 500.672714] alg: No test for adler32 (adler32-zlib) [ 501.533746] Lustre: Lustre: Build Version: 2.17.58_3_g671df26 [ 501.620051] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 503.199125] Key type lgssc registered [ 503.518887] Lustre: Echo OBD driver; http://www.lustre.org/ [ 514.289091] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 514.295247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 515.385646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 515.396289] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 515.432592] Lustre: lustre-MDT0000: new disk, initializing [ 515.455253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 515.461121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 516.787552] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 520.910756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 520.949850] Lustre: 45818:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 520.970812] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 520.972858] Lustre: Skipped 1 previous similar message [ 521.004734] Lustre: lustre-MDT0001: new disk, initializing [ 521.036992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 521.048461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 521.051969] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 522.439383] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 526.084439] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 527.837966] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 530.095949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 530.203205] Lustre: lustre-OST0000: new disk, initializing [ 530.205099] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 530.208474] Lustre: 47782:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 530.240857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 532.103244] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 535.541921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 535.544787] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 535.573536] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 535.737063] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 537.151428] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 537.220878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 538.496768] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 538.563038] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 540.641128] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 540.642330] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 540.643439] LustreError: Skipped 1 previous similar message [ 540.643882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 540.648206] Lustre: Skipped 1 previous similar message [ 541.266274] Lustre: server umount lustre-OST0000 complete [ 545.761267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 545.762151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 545.765341] Lustre: Skipped 1 previous similar message [ 545.769990] Lustre: Skipped 2 previous similar messages [ 548.631033] Lustre: server umount lustre-MDT0000 complete [ 549.834268] LustreError: 45810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788738347 with bad export cookie 17467779394103988587 [ 549.837525] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 549.839345] LustreError: 45810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 549.956723] Lustre: server umount lustre-MDT0001 complete [ 552.869288] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 19:45:50 (1788738350) [ 554.973907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 555.126959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 556.336630] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 559.016043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 560.494136] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 562.831490] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 564.543769] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 566.652577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 566.755325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 566.757332] Lustre: Skipped 1 previous similar message [ 568.556662] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 570.893092] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 583.364874] LustreError: 52790:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 583.650212] Lustre: Modifying parameter general.timeout=1 in log params [ 584.743275] Lustre: Modifying parameter general.timeout=5 in log params [ 584.745569] Lustre: Skipped 3 previous similar messages [ 587.003877] Lustre: Modifying parameter general.timeout=13 in log params [ 587.006030] Lustre: Skipped 7 previous similar messages [ 591.135221] Lustre: Modifying parameter general.timeout=28 in log params [ 591.136951] Lustre: Skipped 14 previous similar messages [ 599.192952] Lustre: Modifying parameter general.timeout=57 in log params [ 599.194434] Lustre: Skipped 28 previous similar messages [ 615.266050] Lustre: Modifying parameter general.timeout=114 in log params [ 615.269993] Lustre: Skipped 56 previous similar messages [ 623.452374] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 19:47:00 (1788738420) [ 624.427630] Lustre: 59679:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 625.641077] Lustre: 59885:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 625.646463] Lustre: 59885:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 635.883627] Lustre: 61531:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 635.886833] Lustre: 61531:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 3 previous similar messages [ 635.890461] LustreError: 61531:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 637.219881] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 19:47:14 (1788738434) [ 640.991938] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 640.994084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 640.997599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 642.848103] Lustre: server umount lustre-MDT0000 complete [ 644.091165] LustreError: 50266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788738441 with bad export cookie 17467779394103989469 [ 644.094092] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 644.094855] LustreError: 50266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 644.101171] LustreError: 50232:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 646.255711] Lustre: server umount lustre-MDT0001 complete [ 671.303363] Lustre: server umount lustre-OST0000 complete [ 680.078159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 680.169987] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 680.179958] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 680.184439] Lustre: Skipped 38 previous similar messages [ 680.269831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 681.609867] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 684.456139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 684.490404] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 684.500630] Lustre: 64059:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 686.034897] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 688.407270] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 690.250624] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 692.458565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 692.529574] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 694.532326] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 696.904726] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 702.807346] Lustre: 66622:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 707.554484] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 707.554598] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 707.555409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 707.555415] Lustre: Skipped 3 previous similar messages [ 707.556951] LustreError: Skipped 1 previous similar message [ 707.561576] Lustre: Skipped 3 previous similar messages [ 710.732983] Lustre: server umount lustre-OST0000 complete [ 712.673123] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 712.673721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 712.678637] Lustre: Skipped 1 previous similar message [ 712.684423] Lustre: Skipped 2 previous similar messages [ 718.102520] Lustre: server umount lustre-MDT0000 complete [ 719.336368] LustreError: 64053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788738516 with bad export cookie 17467779394103991751 [ 719.338960] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 719.343065] LustreError: 64053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 719.457346] Lustre: server umount lustre-MDT0001 complete [ 723.168995] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 724.225590] Key type lgssc unregistered [ 724.359382] LNet: 67832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 724.362372] LNetError: 67832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 724.370660] LNet: Removed LNI 192.168.203.142@tcp [ 724.701413] Key type .llcrypt unregistered [ 724.702845] Key type ._llcrypt unregistered [ 733.781676] Key type ._llcrypt registered [ 733.782879] Key type .llcrypt registered [ 733.824316] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 734.099156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 734.108830] alg: No test for adler32 (adler32-zlib) [ 734.950344] Lustre: Lustre: Build Version: 2.17.58_3_g671df26 [ 735.032627] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 736.607104] Key type lgssc registered [ 736.954536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 739.408768] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 739.415951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 740.567437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 741.920147] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 744.608334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 744.796719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 746.103089] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 748.496947] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 750.387319] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 752.649813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 752.864880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 754.798413] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 757.147746] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 770.164427] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 775.648740] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 775.651876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 775.656951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 776.672478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 776.674173] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 776.675156] Lustre: Skipped 1 previous similar message [ 776.678643] Lustre: Skipped 1 previous similar message [ 778.713238] Lustre: server umount lustre-MDT0000 complete [ 779.922480] LustreError: 69198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788738577 with bad export cookie 8749348263708601392 [ 779.926286] LustreError: 69198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 779.926520] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 780.046698] Lustre: server umount lustre-MDT0001 complete [ 790.496507] Lustre: server umount lustre-OST0000 complete [ 799.366529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 799.470504] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 799.482379] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 799.573123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 800.964823] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 803.828972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 803.860798] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 803.869714] Lustre: 74101:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 804.038336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 805.418424] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 807.802288] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 809.575436] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 811.776118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 811.849293] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 813.900667] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 816.246247] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 822.065981] Lustre: 76674:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 822.682070] Lustre: Modifying parameter general.jobid_var=disable in log params [ 826.850503] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 826.851146] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 826.857523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 826.859502] Lustre: Skipped 1 previous similar message [ 830.030627] Lustre: server umount lustre-OST0000 complete [ 831.968494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 831.969029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 831.972158] Lustre: Skipped 2 previous similar messages [ 837.088318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 837.091403] Lustre: Skipped 2 previous similar messages [ 837.401696] Lustre: server umount lustre-MDT0000 complete [ 838.663396] LustreError: 74093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788738636 with bad export cookie 8749348263708602204 [ 838.666883] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 838.667782] LustreError: 74093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 838.777168] Lustre: server umount lustre-MDT0001 complete [ 842.495975] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 843.554807] Key type lgssc unregistered [ 843.691423] LNet: 77878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.693933] LNetError: 77878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 843.703376] LNet: Removed LNI 192.168.203.142@tcp [ 844.031393] Key type .llcrypt unregistered [ 844.032609] Key type ._llcrypt unregistered [ 852.934632] Key type ._llcrypt registered [ 852.935666] Key type .llcrypt registered [ 852.972453] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 853.298339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 853.341031] alg: No test for adler32 (adler32-zlib) [ 854.206295] Lustre: Lustre: Build Version: 2.17.58_3_g671df26 [ 854.305827] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 855.887103] Key type lgssc registered [ 856.276262] Lustre: Echo OBD driver; http://www.lustre.org/ [ 858.968144] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 858.977698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 860.175632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 861.605059] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 864.394974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 864.586069] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 865.954611] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 868.313656] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 870.153871] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 872.403058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 872.585351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 874.454290] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 876.772047] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 889.375024] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 889.975093] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 889.977241] Lustre: Skipped 1 previous similar message [ 891.155388] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 891.157261] Lustre: Skipped 3 previous similar messages [ 893.183215] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 893.185150] Lustre: Skipped 6 previous similar messages [ 897.216248] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 897.218502] Lustre: Skipped 13 previous similar messages [ 905.340608] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 905.343019] Lustre: Skipped 27 previous similar messages [ 921.001826] Lustre: 87348:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 921.873639] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 19:51:59 (1788738719) [ 922.402811] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 922.965921] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 19:52:00 (1788738720) [ 926.176466] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 926.180210] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 926.181683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 926.184235] Lustre: Skipped 1 previous similar message [ 929.999629] Lustre: server umount lustre-OST0000 complete [ 931.296655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 931.296738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 931.297411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 931.297415] Lustre: Skipped 1 previous similar message [ 931.300567] Lustre: Skipped 1 previous similar message [ 931.308075] LustreError: Skipped 1 previous similar message [ 936.416111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 936.418102] Lustre: Skipped 2 previous similar messages [ 937.379308] Lustre: server umount lustre-MDT0000 complete [ 938.601916] LustreError: 79245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788738736 with bad export cookie 8605559689938531672 [ 938.605950] LustreError: 79245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 938.606252] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 938.738530] Lustre: server umount lustre-MDT0001 complete [ 942.232495] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 943.184804] Key type lgssc unregistered [ 943.316386] LNet: 88592:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 943.318972] LNetError: 88592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 943.325292] LNet: Removed LNI 192.168.203.142@tcp [ 943.623065] Key type .llcrypt unregistered [ 943.624570] Key type ._llcrypt unregistered [ 944.972817] Key type ._llcrypt registered [ 944.973864] Key type .llcrypt registered [ 945.007928] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 946.372878] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules [ 946.783046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 946.813110] alg: No test for adler32 (adler32-zlib) [ 947.668252] Lustre: Lustre: Build Version: 2.17.58_3_g671df26 [ 947.751831] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 949.327182] Key type lgssc registered [ 949.692582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 950.068635] LustreError: 89740:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 957.786453] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 960.731626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 971.384838] LustreError: 89740:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 971.393354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 972.553831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 973.837136] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 976.490688] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 19:52:53 (1788738773) [ 978.026234] LustreError: 90797:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 978.029922] LustreError: 90797:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 978.032426] LustreError: 90797:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 5, retries 0, failed: rc = -5 [ 978.154706] Lustre: server umount lustre-MDT0000 complete [ 982.518210] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 983.518493] Key type lgssc unregistered [ 983.645340] LNet: 92290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.647963] LNetError: 92290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 983.657274] LNet: Removed LNI 192.168.203.142@tcp [ 983.954224] Key type .llcrypt unregistered [ 983.955374] Key type ._llcrypt unregistered [ 990.963461] Key type ._llcrypt registered [ 990.964479] Key type .llcrypt registered [ 991.000759] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 991.423513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 991.431389] alg: No test for adler32 (adler32-zlib) [ 992.287491] Lustre: Lustre: Build Version: 2.17.58_3_g671df26 [ 992.382495] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 993.959211] Key type lgssc registered [ 994.317300] Lustre: Echo OBD driver; http://www.lustre.org/ [ 996.916885] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 996.922873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 998.077048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 999.411894] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1002.265652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1002.480610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1003.748748] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1006.014916] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1008.022091] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1010.188779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1010.388683] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1012.216595] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1014.511983] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1045.548405] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 19:54:02 (1788738842) [ 1046.108255] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1046.671443] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 19:54:03 (1788738843) [ 1048.545141] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1048.546506] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1048.547401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1048.557109] Lustre: Skipped 2 previous similar messages [ 1053.664445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1053.666867] Lustre: Skipped 4 previous similar messages [ 1053.725699] Lustre: server umount lustre-MDT0000 complete [ 1054.893150] LustreError: 93464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788738852 with bad export cookie 13212039737702991526 [ 1054.896909] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1054.897247] LustreError: 93464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1055.006937] Lustre: server umount lustre-MDT0001 complete [ 1065.594343] Lustre: server umount lustre-OST0000 complete [ 1069.225228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1069.379789] LustreError: 97767:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1069.386422] LustreError: 97767:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1069.433637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1070.781567] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1073.720802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1073.968288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1075.409906] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1077.967443] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1079.896156] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1084.345563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1084.387183] LustreError: 99855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1084.440578] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1084.444269] LustreError: 97757:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1084.459447] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1084.462698] LustreError: 99848:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 1084.466428] LustreError: 99848:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 1084.468694] LustreError: 99848:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1084.496127] Lustre: server umount lustre-OST0000 complete [ 1084.497821] LustreError: 99848:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1087.313051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1087.382592] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1087.385152] LustreError: 97755:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1087.399454] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1087.402697] LustreError: 100313:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 1087.404796] LustreError: 100313:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 1087.406659] LustreError: 100313:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1087.429881] Lustre: server umount lustre-OST0000 complete [ 1087.432580] LustreError: 100313:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1090.482488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1090.544442] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1090.546213] Lustre: Found index 0 for lustre-OST0000, updating log [ 1090.548307] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1090.569197] Lustre: lustre-OST0000: new disk, initializing [ 1090.570866] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1090.573298] Lustre: 100842:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1090.662216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1092.531179] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1093.700173] Lustre: Failing over lustre-OST0000 [ 1093.743416] Lustre: server umount lustre-OST0000 complete [ 1099.744400] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1099.750065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1099.759896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1101.084176] Lustre: server umount lustre-MDT0000 complete [ 1102.281055] LustreError: 97749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788738899 with bad export cookie 13212039737702994256 [ 1102.283755] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1102.284484] LustreError: 97749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1102.393244] Lustre: server umount lustre-MDT0001 complete [ 1105.080900] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 19:55:02 (1788738902) [ 1110.739302] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1115.281569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1115.454291] LustreError: 103180:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1115.487606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1116.892283] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1120.175922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1121.805248] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1123.165153] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1126.350627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1128.529802] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1131.844249] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1131.886751] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1131.927828] Lustre: lustre-OST0001: new disk, initializing [ 1131.931432] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1131.936390] Lustre: 105498:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1134.081283] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1135.938627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1135.942368] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1135.975438] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1143.655204] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1152.959148] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1154.857007] hrtimer: interrupt took 1937800 ns [ 1161.704494] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1161.710606] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1161.712660] Lustre: Skipped 1 previous similar message [ 1161.715402] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1161.715411] Lustre: Skipped 2 previous similar messages [ 1165.820312] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.42@tcp (stopping) [ 1165.824217] Lustre: Skipped 2 previous similar messages [ 1167.555138] Lustre: server umount lustre-MDT0001 complete [ 1170.907831] LustreError: 105169:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1176.036393] LustreError: 103911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1176.084278] LustreError: 103911:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1184.638146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1184.696963] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1184.701905] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1184.738763] Lustre: 103170:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1185.206755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1185.215526] Lustre: Skipped 3 previous similar messages [ 1186.756079] LustreError: 107076:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1186.766342] LustreError: 107076:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1186.779164] LustreError: 103173:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1187.055273] LustreError: 107081:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.142@tcp: cfg command failed: rc = -17 [ 1187.064753] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1187.064753] [ 1190.383204] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1192.571788] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1203.678972] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 19:56:39 (1788738999) [ 1241.902947] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1267.143160] Lustre: 109138:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1288.571188] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1292.128739] Lustre: 110273:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1369.567777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1369.568896] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1369.570872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1369.570879] Lustre: Skipped 3 previous similar messages [ 1369.621611] Lustre: Skipped 5 previous similar messages [ 1373.590425] Lustre: server umount lustre-MDT0000 complete [ 1374.691466] LustreError: 105169:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1374.717146] LustreError: 105169:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 1377.060974] LustreError: 103161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788739174 with bad export cookie 13212039737702995166 [ 1377.064211] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1377.075318] LustreError: 103161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1377.422334] Lustre: server umount lustre-MDT0001 complete [ 1390.946388] Lustre: server umount lustre-OST0000 complete [ 1405.699748] Lustre: server umount lustre-OST0001 complete [ 1414.124849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1429.936271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1440.207974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1451.289694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1473.071274] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1483.461559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1483.477462] Lustre: lustre-MDT0000: reset Object Index mappings [ 1483.919762] LustreError: 114242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1484.011280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1489.476907] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1500.033063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1500.050622] Lustre: lustre-MDT0001: reset Object Index mappings [ 1507.124706] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1510.250616] Lustre: 115390:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1516.987226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1524.459311] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1525.801854] LustreError: 115742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1525.816287] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 1525.838743] LustreError: 115742:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1528.900693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:350 to 0x280000401:386) [ 1534.886862] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1540.737907] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1541.111033] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 1541.116609] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 1547.341226] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1550.720607] Lustre: 117267:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1589.350625] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 20:03:05 (1788739385) [ 1618.400335] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1618.403456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1618.411135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1618.415020] Lustre: Skipped 3 previous similar messages [ 1624.454763] Lustre: server umount lustre-MDT0000 complete [ 1628.136413] LustreError: 114977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1628.149864] LustreError: 114977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 1629.747932] LustreError: 114223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788739427 with bad export cookie 13212039737703041954 [ 1629.757881] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1633.249264] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1633.264669] Lustre: Skipped 4 previous similar messages [ 1636.236387] Lustre: server umount lustre-MDT0001 complete [ 1655.271565] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1655.480163] Lustre: server umount lustre-OST0000 complete [ 1674.208161] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1674.457291] Lustre: server umount lustre-OST0001 complete [ 1681.662882] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 1694.078137] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1752.762996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1753.080677] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1753.137847] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1753.255513] Lustre: lustre-MDT0000: new disk, initializing [ 1753.351047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1753.354078] Lustre: Skipped 3 previous similar messages [ 1753.380917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1758.807954] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1768.949301] Lustre: server umount lustre-MDT0000 complete [ 1781.541853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1782.008415] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1782.041564] Lustre: 121545:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1782.066489] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1787.156288] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1792.574882] Lustre: server umount lustre-MDT0000 complete [ 1803.540948] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 20:06:39 (1788739599) [ 1806.426843] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1809.690397] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 20:06:44 (1788739604) [ 1829.243670] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1842.812147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1847.378137] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1857.274846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1857.464179] Lustre: 123598:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1857.577483] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1857.586471] Lustre: Skipped 1 previous similar message [ 1857.699587] Lustre: lustre-MDT0001: new disk, initializing [ 1857.976070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1858.001870] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1863.097674] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1868.629272] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1875.458963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1875.765374] Lustre: lustre-OST0000: new disk, initializing [ 1875.770493] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1875.789916] Lustre: 125173:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1877.110050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1877.125444] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1877.283669] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1883.487588] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1895.652913] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1895.943802] Lustre: lustre-OST0001: new disk, initializing [ 1895.951893] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1895.979168] Lustre: 126044:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1898.049219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1898.060724] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1898.192433] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1906.877221] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1918.851527] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1925.318754] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2319.757477] Lustre: 132463:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2319.767539] Lustre: 132463:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2319.772932] Lustre: 132463:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2319.779050] Lustre: 132463:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2319.785594] Lustre: 132463:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2319.789807] Lustre: 132463:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2552.393101] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 20:19:07 (1788740347) [ 2569.192489] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2569.199523] Lustre: Skipped 3 previous similar messages [ 2569.199737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2569.211510] Lustre: Skipped 11 previous similar messages [ 2573.004678] Lustre: server umount lustre-MDT0000 complete [ 2574.307257] LustreError: 123605:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2574.337898] LustreError: 123605:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2576.426409] LustreError: 124346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788740374 with bad export cookie 13212039737703106508 [ 2576.428467] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2576.451361] LustreError: 124346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2579.424502] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2579.427525] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2579.444362] Lustre: Skipped 1 previous similar message [ 2582.810849] Lustre: server umount lustre-MDT0001 complete [ 2591.677903] Lustre: server umount lustre-OST0000 complete [ 2601.205596] Lustre: server umount lustre-OST0001 complete [ 2607.128704] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 2615.728736] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2649.970417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2650.246552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2650.281152] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2650.327643] Lustre: lustre-MDT0000: new disk, initializing [ 2650.426102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2650.430671] Lustre: Skipped 5 previous similar messages [ 2650.443821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2655.261884] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2664.637949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2664.722695] Lustre: 135506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2664.804590] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2664.806905] Lustre: Skipped 1 previous similar message [ 2664.890874] Lustre: lustre-MDT0001: new disk, initializing [ 2665.020267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2665.036959] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2669.114696] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2677.610826] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2682.927373] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2688.644265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2688.884740] Lustre: lustre-OST0000: new disk, initializing [ 2688.891620] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2688.897880] Lustre: 137477:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2689.011772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2689.023065] Lustre: Skipped 1 previous similar message [ 2690.572291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2690.584133] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2690.690786] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2694.637618] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2702.630526] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2710.533878] Lustre: lustre-MDD0000: changelog on [ 2712.843232] Lustre: lustre-MDD0001: changelog on [ 4480.152552] Lustre: lustre-MDD0001: changelog off [ 4481.691349] Lustre: 136668:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 4482.654684] Lustre: lustre-MDD0000: changelog off [ 4491.746338] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4491.756908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4491.761565] Lustre: Skipped 3 previous similar messages [ 4495.379706] Lustre: server umount lustre-OST0000 complete [ 4501.984557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4502.000923] Lustre: Skipped 1 previous similar message [ 4502.005565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4502.009458] Lustre: Skipped 2 previous similar messages [ 4507.107606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4507.118962] Lustre: Skipped 1 previous similar message [ 4512.229546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4512.239431] Lustre: Skipped 1 previous similar message [ 4514.271410] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4514.756459] Lustre: server umount lustre-MDT0000 complete [ 4517.349061] LustreError: 135511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4517.374846] LustreError: 135511:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4518.611704] LustreError: 135541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788742316 with bad export cookie 13212039737703924549 [ 4518.627892] LustreError: 135541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4518.629996] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4519.077278] Lustre: server umount lustre-MDT0001 complete [ 4528.981538] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 4531.519407] Key type lgssc unregistered [ 4531.809953] LNet: 146596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4531.814763] LNetError: 146596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4531.837522] LNet: Removed LNI 192.168.203.142@tcp [ 4532.736247] Key type .llcrypt unregistered [ 4532.743288] Key type ._llcrypt unregistered [ 4553.973178] Key type ._llcrypt registered [ 4553.975198] Key type .llcrypt registered [ 4554.083270] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 4569.549618] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 4570.407391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4570.427349] alg: No test for adler32 (adler32-zlib) [ 4571.598234] Lustre: Lustre: Build Version: 2.17.58_3_g671df26 [ 4571.970543] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 4573.632481] Key type lgssc registered [ 4574.542502] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4609.744568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4609.796304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4611.142941] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4611.196360] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4611.325229] Lustre: lustre-MDT0000: new disk, initializing [ 4611.429875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4611.459703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4616.927458] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4628.065419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4628.197893] Lustre: 149797:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4628.260226] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4628.270312] Lustre: Skipped 1 previous similar message [ 4628.353265] Lustre: lustre-MDT0001: new disk, initializing [ 4628.461393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4628.504415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4628.524240] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4632.996135] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4641.401050] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4646.796298] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4653.663738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4654.016847] Lustre: lustre-OST0000: new disk, initializing [ 4654.020646] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4654.026072] Lustre: 151765:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4654.113216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4659.821210] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4661.837070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4661.849512] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4661.929549] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4670.088319] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4676.380765] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_module obdecho/obdecho [ 4678.300667] LustreError: 152767:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 4678.308319] LustreError: 152767:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 4680.244746] LustreError: 152863:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 4692.961966] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4692.970111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4692.986716] Lustre: Skipped 1 previous similar message [ 4693.012630] Lustre: Skipped 2 previous similar messages [ 4698.082846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4698.099314] Lustre: Skipped 2 previous similar messages [ 4699.071310] Lustre: server umount lustre-MDT0000 complete [ 4702.755585] LustreError: 149788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788742500 with bad export cookie 4440001180642593024 [ 4702.762545] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4702.767478] LustreError: 149788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4703.200704] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4703.218137] Lustre: Skipped 1 previous similar message [ 4703.328575] Lustre: server umount lustre-MDT0001 complete [ 4709.447968] Lustre: server umount lustre-OST0000 complete [ 4727.926543] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 4736.522900] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 4767.255992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4767.434131] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4767.452555] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4767.498863] Lustre: lustre-MDT0000: new disk, initializing [ 4767.546454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4767.557742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4770.716989] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4778.881238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4778.973659] Lustre: 156578:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4779.039078] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4779.041878] Lustre: Skipped 1 previous similar message [ 4779.102994] Lustre: lustre-MDT0001: new disk, initializing [ 4779.173592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4779.208576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4779.215759] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4782.533858] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4789.926588] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4793.787819] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4797.957214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4798.184947] Lustre: lustre-OST0000: new disk, initializing [ 4798.188759] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4798.194524] Lustre: 158546:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4798.272851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4802.565563] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4805.664271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4805.677775] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4805.752495] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4808.914064] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4812.330986] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4812.518476] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4816.053165] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4816.198118] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4820.968825] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4820.969953] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4820.973193] LustreError: Skipped 1 previous similar message [ 4820.990849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4821.001864] Lustre: Skipped 1 previous similar message [ 4823.709487] Lustre: server umount lustre-OST0000 complete [ 4830.688852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4830.694528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4830.712418] Lustre: Skipped 1 previous similar message [ 4830.722655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4832.832691] Lustre: server umount lustre-MDT0000 complete [ 4835.785685] LustreError: 156568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788742633 with bad export cookie 4440001180642593983 [ 4835.787312] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4835.799887] LustreError: 156568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4836.067768] Lustre: server umount lustre-MDT0001 complete [ 4839.248512] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 20:57:15 (1788742635) [ 4840.629474] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 4842.038979] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 20:57:18 (1788742638) [ 4847.790755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4848.374871] LustreError: 160953:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4848.396706] LustreError: 160953:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4848.459419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4852.693265] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4853.730890] LustreError: 160954:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4858.853344] LustreError: 160953:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4859.713959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4863.242691] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4869.013432] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4872.787259] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4877.700841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4883.033242] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4888.831132] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4899.815331] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4899.823820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4899.831545] Lustre: Skipped 3 previous similar messages [ 4899.846016] Lustre: Skipped 5 previous similar messages [ 4900.796187] Lustre: server umount lustre-MDT0000 complete [ 4903.024445] LustreError: 160935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788742700 with bad export cookie 4440001180642594865 [ 4903.026572] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4903.032384] LustreError: 160935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4904.924083] LustreError: 160950:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4904.950542] LustreError: 160950:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4908.960468] LustreError: 160948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4908.960508] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4908.983865] Lustre: Skipped 2 previous similar messages [ 4909.462844] Lustre: server umount lustre-MDT0001 complete [ 4914.284970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4925.113345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4925.273665] LustreError: 164791:0:(llog.c:1655:llog_backup()) MGC192.168.203.142@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 4925.281488] Lustre: 164791:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.142@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 4929.560895] LustreError: 164797:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4945.882875] LustreError: 164798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4945.894049] LustreError: 164798:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 4981.723040] LustreError: 164797:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4981.750860] LustreError: 164797:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [ 4986.335342] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4986.342350] Lustre: 164791:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.142@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 5027.327318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5027.347885] Lustre: Skipped 2 previous similar messages [ 5031.177432] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5032.425296] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5037.592489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5047.260924] LustreError: 164798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5047.299101] LustreError: 164798:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 42 previous similar messages [ 5047.781878] LustreError: 148014:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003646f37a x1875632318436992/t0(0) o250->MGC192.168.203.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5052.929508] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5053.425380] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5059.160558] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5060.540661] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5066.624606] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5067.970552] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5084.130036] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5084.132380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5084.133389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5084.138992] Lustre: Skipped 2 previous similar messages [ 5084.806933] Lustre: server umount lustre-MDT0000 complete [ 5087.233899] LustreError: 161692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788742884 with bad export cookie 4440001180642601326 [ 5087.239694] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5087.239702] LustreError: 161692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5087.397985] Lustre: server umount lustre-MDT0001 complete [ 5100.398136] Lustre: server umount lustre-OST0000 complete [ 5121.347765] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 5133.200102] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5167.252517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5167.524738] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5167.560373] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5167.668244] Lustre: lustre-MDT0000: new disk, initializing [ 5167.767207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5167.773366] Lustre: Skipped 1 previous similar message [ 5167.792253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5172.346775] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5181.031688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5181.122538] Lustre: 169906:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5181.376524] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5184.964133] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5193.439803] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5198.474096] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5204.151185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5204.353828] Lustre: lustre-OST0000: new disk, initializing [ 5204.357028] Lustre: Skipped 1 previous similar message [ 5204.360745] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5204.364525] Lustre: Skipped 2 previous similar messages [ 5204.369244] Lustre: 171875:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5209.323335] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5213.746093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5213.757733] Lustre: Skipped 1 previous similar message [ 5213.775569] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5213.869268] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5218.530145] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5223.660123] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5223.875115] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5228.177969] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5228.403728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5234.658598] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5234.660320] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5234.664065] LustreError: Skipped 1 previous similar message [ 5234.669898] Lustre: Skipped 1 previous similar message [ 5234.674054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5234.681155] Lustre: Skipped 3 previous similar messages [ 5244.384930] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5244.527731] Lustre: server umount lustre-OST0000 complete [ 5247.968856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5255.136507] LustreError: 169918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5255.160672] LustreError: 169918:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 5255.945165] Lustre: server umount lustre-MDT0000 complete [ 5258.905838] LustreError: 169898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788743056 with bad export cookie 4440001180642618147 [ 5258.909410] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5258.914813] LustreError: 169898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5259.196600] Lustre: server umount lustre-MDT0001 complete [ 5265.735673] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 21:04:22 (1788743062) [ 5270.888546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5274.942928] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5282.479389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5288.901432] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5297.811869] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5304.816284] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5312.443178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5312.749203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5312.754488] Lustre: Skipped 4 previous similar messages [ 5318.153832] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5326.768791] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5343.025490] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 21:05:38 (1788743138) [ 5349.925425] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5349.931385] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5349.949569] Lustre: Skipped 2 previous similar messages [ 5349.955629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5349.960650] Lustre: Skipped 5 previous similar messages [ 5351.779936] Lustre: server umount lustre-OST0000 complete [ 5362.700274] Lustre: server umount lustre-MDT0000 complete [ 5367.717207] LustreError: 174270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788743165 with bad export cookie 4440001180642619029 [ 5367.731500] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5367.733907] LustreError: 174270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5368.029539] Lustre: server umount lustre-MDT0001 complete [ 5377.233076] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 5379.685856] Key type lgssc unregistered [ 5379.950835] LNet: 178017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5379.961150] LNetError: 178017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5379.998453] LNet: Removed LNI 192.168.203.142@tcp [ 5380.659214] Key type .llcrypt unregistered [ 5380.661029] Key type ._llcrypt unregistered [ 5400.881145] Key type ._llcrypt registered [ 5400.883192] Key type .llcrypt registered [ 5400.988501] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5401.833526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5402.035901] alg: No test for adler32 (adler32-zlib) [ 5403.243237] Lustre: Lustre: Build Version: 2.17.58_3_g671df26 [ 5403.649490] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 5405.391168] Key type lgssc registered [ 5406.402325] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5416.059346] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5416.074627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5431.652185] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5436.767364] LustreError: 179259:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.142@tcp: failed processing log, type 4: rc = -110 [ 5441.887687] LustreError: 179259:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5441.901685] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 5441.907965] LustreError: MGC192.168.203.142@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 5441.917473] LustreError: 179259:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 5441.925918] LustreError: 179259:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -22 [ 5441.932718] LustreError: 179259:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 5441.980777] Lustre: server umount lustre-OST0000 complete [ 5441.985701] LustreError: 179259:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5448.828029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5450.184350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5453.773137] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5459.427088] LustreError: 179758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5459.468449] LustreError: 179758:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5461.489640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5461.965278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5466.830834] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5474.057608] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5479.157238] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5485.250382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5485.400440] LustreError: 181573:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5485.408500] Lustre: 181573:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.142@tcp: invalid local config log lustre-OST0000: rc = -22 [ 5485.527350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5490.143722] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5496.025684] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5497.829795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5497.841496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5497.854178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5499.873140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5499.875823] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5503.935817] Lustre: server umount lustre-MDT0000 complete [ 5504.994862] LustreError: 179759:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5505.011074] LustreError: 179759:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5507.089355] LustreError: 179739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788743304 with bad export cookie 3199049204554527350 [ 5507.097236] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5507.100423] LustreError: 179739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5507.333866] Lustre: server umount lustre-MDT0001 complete [ 5520.635790] Lustre: server umount lustre-OST0000 complete [ 5529.732866] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 5539.899965] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5575.194570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5575.428112] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5575.455961] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5575.534500] Lustre: lustre-MDT0000: new disk, initializing [ 5575.606403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5575.632100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5580.636806] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5592.727994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5592.973534] Lustre: 185094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5593.085680] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5593.095955] Lustre: Skipped 1 previous similar message [ 5593.241399] Lustre: lustre-MDT0001: new disk, initializing [ 5593.341860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5593.382815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5593.393836] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5597.990533] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5606.548911] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5611.294616] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5618.126317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5618.382464] Lustre: lustre-OST0000: new disk, initializing [ 5618.387408] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5618.392774] Lustre: 187063:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5618.484325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5625.313257] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5625.431742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5625.455153] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5625.563519] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5636.285711] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5641.966610] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5642.188711] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5645.721391] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5645.884001] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5651.427048] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5651.429510] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5651.437978] Lustre: Skipped 1 previous similar message [ 5651.438886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5651.447526] LustreError: Skipped 1 previous similar message [ 5651.449892] Lustre: Skipped 2 previous similar messages [ 5653.692166] Lustre: server umount lustre-OST0000 complete [ 5656.557052] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5656.561365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5656.566466] Lustre: Skipped 2 previous similar messages [ 5661.600537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5661.607611] Lustre: Skipped 4 previous similar messages [ 5662.864191] Lustre: server umount lustre-MDT0000 complete [ 5666.784020] LustreError: 185106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5667.136844] LustreError: 185088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788743464 with bad export cookie 3199049204554528071 [ 5667.149335] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5667.168729] LustreError: 185088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5667.409066] Lustre: server umount lustre-MDT0001 complete [ 5676.160371] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 21:11:12 (1788743472) [ 5692.612404] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 5695.137943] Key type lgssc unregistered [ 5695.340109] LNet: 190009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5695.346273] LNetError: 190009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5695.368926] LNet: Removed LNI 192.168.203.142@tcp [ 5696.066700] Key type .llcrypt unregistered [ 5696.068576] Key type ._llcrypt unregistered [ 5712.744262] Key type ._llcrypt registered [ 5712.750335] Key type .llcrypt registered [ 5712.926460] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5713.976230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5714.033897] alg: No test for adler32 (adler32-zlib) [ 5715.260046] Lustre: Lustre: Build Version: 2.17.58_3_g671df26 [ 5715.594938] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 5717.250812] Key type lgssc registered [ 5718.673802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5726.782789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5726.828692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5728.465313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5733.934349] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5738.979649] LustreError: 191224:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5738.996426] LustreError: 191224:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5742.825763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5743.102449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5748.077295] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5756.274526] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5762.324528] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5769.070655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5769.242290] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5769.244680] LustreError: 193089:0:(llog.c:1696:llog_backup()) MGC192.168.203.142@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5769.249484] Lustre: 193089:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.142@tcp: can't backup local config lustre-OST0000: rc = -28 [ 5769.361549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5775.184862] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5782.763478] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5785.057824] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5785.059527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5785.088245] Lustre: Skipped 2 previous similar messages [ 5785.105445] Lustre: Skipped 2 previous similar messages [ 5788.194565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5791.034494] Lustre: server umount lustre-MDT0000 complete [ 5793.249921] LustreError: 191225:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5793.269909] LustreError: 191225:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5795.126265] LustreError: 191204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788743592 with bad export cookie 16239583909777050532 [ 5795.132815] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5795.133310] LustreError: 191204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5796.513881] LustreError: 191954:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000099a701bf x1875633517915648/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5796.864970] Lustre: server umount lustre-MDT0001 complete [ 5811.406658] Lustre: server umount lustre-OST0000 complete [ 5821.556562] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 5833.817393] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5869.405043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5869.632367] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5869.658933] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5869.739948] Lustre: lustre-MDT0000: new disk, initializing [ 5869.817815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5869.834473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5874.151278] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5883.236118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5883.348769] Lustre: 196616:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5883.386974] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5883.388567] Lustre: Skipped 1 previous similar message [ 5883.391272] LustreError: 197396:0:(llog.c:1696:llog_backup()) MGC192.168.203.142@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 5883.397133] LustreError: 197396:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 5883.400825] Lustre: 197396:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.142@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 5883.416411] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5883.420158] Lustre: Skipped 1 previous similar message [ 5883.489213] Lustre: lustre-MDT0001: new disk, initializing [ 5883.588804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5883.618606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5883.627414] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5887.683918] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5895.615720] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5900.996859] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5907.089917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5907.376436] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5907.379385] Lustre: Skipped 1 previous similar message [ 5907.383643] LustreError: 198558:0:(llog.c:1696:llog_backup()) MGC192.168.203.142@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5907.392096] LustreError: 198558:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 5907.405851] Lustre: 198558:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.142@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 5907.419722] Lustre: 198558:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 5907.468695] Lustre: lustre-OST0000: new disk, initializing [ 5907.474364] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5907.494666] Lustre: 198581:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5907.595878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5913.169785] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5916.244148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5916.255206] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5916.390199] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5922.552560] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5926.274408] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5926.467581] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5929.948242] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5930.113782] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5932.000297] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5932.007323] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5932.011955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5932.014444] Lustre: Skipped 1 previous similar message [ 5932.032922] Lustre: Skipped 3 previous similar messages [ 5937.135897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5937.143663] Lustre: Skipped 1 previous similar message [ 5937.835319] Lustre: server umount lustre-OST0000 complete [ 5942.255572] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5942.285240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5942.299772] Lustre: Skipped 1 previous similar message [ 5947.730066] Lustre: server umount lustre-MDT0000 complete [ 5950.584438] LustreError: 196606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788743748 with bad export cookie 16239583909777051260 [ 5950.602489] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5950.608836] LustreError: 196606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5950.892951] Lustre: server umount lustre-MDT0001 complete [ 5957.809639] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 21:15:54 (1788743754) [ 5979.589046] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5989.644655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5990.224377] LustreError: 201726:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5990.301800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5994.545443] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5995.488867] LustreError: 201727:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5999.593489] LustreError: 201726:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6002.036582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6007.557550] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6010.988879] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6017.109483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6024.012190] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6032.568190] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6032.946661] Lustre: lustre-OST0001: new disk, initializing [ 6032.955698] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6032.970229] Lustre: 204037:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6033.074523] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6033.081991] Lustre: Skipped 2 previous similar messages [ 6039.405623] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6040.707311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6040.719574] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6040.875563] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6049.632671] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6057.368515] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6059.303476] Lustre: DEBUG MARKER: ADD OST3 [ 6065.871736] loop: module loaded [ 6065.976713] loop0: detected capacity change from 0 to 204800000 [ 6066.176250] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6066.185521] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6066.197494] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6072.847279] loop0: detected capacity change from 0 to 204800000 [ 6073.306277] Lustre: DEBUG MARKER: STOP OST3 [ 6079.083507] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6079.289139] Lustre: lustre-OST0003: new disk, initializing [ 6079.294705] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 6079.299847] Lustre: 206061:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6080.725295] Lustre: *** cfs_fail_loc=2109, val=2*** [ 6080.855774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 6080.863772] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 6080.896502] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 6082.597918] Lustre: Failing over lustre-OST0003 [ 6082.713600] Lustre: server umount lustre-OST0003 complete [ 6082.783065] LustreError: 206209:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 6085.490615] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6086.115926] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6086.119908] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6086.121884] LustreError: Skipped 1 previous similar message [ 6086.127664] Lustre: Skipped 1 previous similar message [ 6086.129327] LustreError: 203215:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6094.633714] Lustre: DEBUG MARKER: START OST3 again [ 6102.576885] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6102.962372] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 6102.975741] Lustre: Skipped 1 previous similar message [ 6102.988654] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 6104.438296] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 6104.794613] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 6104.802289] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6109.480944] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6131.063282] Lustre: Failing over lustre-OST0003 [ 6131.296621] Lustre: server umount lustre-OST0003 complete [ 6133.741192] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6133.764921] Lustre: Skipped 2 previous similar messages [ 6133.766645] LustreError: 204465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6133.796652] LustreError: 204465:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 6151.136437] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6151.146036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6151.160768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6151.166228] Lustre: Skipped 2 previous similar messages [ 6155.862358] Lustre: server umount lustre-MDT0000 complete [ 6159.158400] LustreError: 202862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788743956 with bad export cookie 16239583909777052142 [ 6159.169170] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6159.174438] LustreError: 202862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6165.565961] Lustre: server umount lustre-MDT0001 complete [ 6169.199026] Lustre: server umount lustre-OST0000 complete [ 6172.995581] Lustre: server umount lustre-OST0001 complete [ 6179.883650] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 6188.563927] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 6224.166110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6224.366292] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6224.425325] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6224.479836] Lustre: lustre-MDT0000: new disk, initializing [ 6224.561401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6229.691513] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6239.684645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6239.809495] Lustre: 211497:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6240.037207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6240.047886] Lustre: Skipped 1 previous similar message [ 6240.087428] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6244.774360] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6254.881680] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6261.422646] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6267.742775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6268.136758] Lustre: lustre-OST0000: new disk, initializing [ 6268.139717] Lustre: Skipped 1 previous similar message [ 6268.144126] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6268.146880] Lustre: Skipped 2 previous similar messages [ 6268.152235] Lustre: 213467:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6269.437483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6269.457431] Lustre: Skipped 1 previous similar message [ 6269.470172] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6269.618166] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6275.136877] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6283.539108] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6287.385270] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6287.590736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6291.494911] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6291.747671] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6295.009248] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6295.020572] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6295.033572] Lustre: Skipped 5 previous similar messages [ 6295.042402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6295.048949] Lustre: Skipped 8 previous similar messages [ 6299.282823] Lustre: server umount lustre-OST0000 complete [ 6308.584104] Lustre: server umount lustre-MDT0000 complete [ 6310.881111] LustreError: 211508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6310.903164] LustreError: 211508:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 6311.775968] LustreError: 211489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788744109 with bad export cookie 16239583909777054431 [ 6311.784637] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6311.788100] LustreError: 211489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6312.049752] Lustre: server umount lustre-MDT0001 complete [ 6315.281558] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 21:21:51 (1788744111) [ 6328.730797] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 6337.272699] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 6373.176038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6373.617957] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6373.644977] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6373.718644] Lustre: lustre-MDT0000: new disk, initializing [ 6373.809550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6378.570137] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6388.629760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6388.733467] Lustre: 218077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6388.957655] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6393.861289] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6405.480433] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6411.145267] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6417.342433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6417.682801] Lustre: 220048:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6419.096231] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6424.162928] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6436.229204] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6442.747677] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6443.013345] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6448.953063] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6449.176929] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6455.268413] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6455.269118] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6455.299599] Lustre: Skipped 3 previous similar messages [ 6455.303985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6455.306559] LustreError: Skipped 2 previous similar messages [ 6455.322882] Lustre: Skipped 4 previous similar messages [ 6457.077359] Lustre: server umount lustre-OST0000 complete [ 6475.231204] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6475.456348] Lustre: server umount lustre-MDT0000 complete [ 6475.744671] LustreError: 218956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6475.764568] LustreError: 218956:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6479.248426] LustreError: 218874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788744276 with bad export cookie 16239583909777055313 [ 6479.250067] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6479.262702] LustreError: 218874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6479.703973] Lustre: server umount lustre-MDT0001 complete [ 6489.342809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6495.333255] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6503.161712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6503.543224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6503.552132] Lustre: Skipped 5 previous similar messages [ 6508.284322] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6514.645777] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6520.390980] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6526.705657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6532.682733] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6539.821881] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6549.475913] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6549.482798] LustreError: Skipped 1 previous similar message [ 6549.485928] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6549.514989] Lustre: Skipped 3 previous similar messages [ 6549.524726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6549.541115] Lustre: Skipped 5 previous similar messages [ 6555.690604] Lustre: server umount lustre-OST0000 complete [ 6565.800453] Lustre: server umount lustre-MDT0000 complete [ 6569.365960] LustreError: 222247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788744366 with bad export cookie 16239583909777056188 [ 6569.400757] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6569.696636] Lustre: server umount lustre-MDT0001 complete [ 6677.684081] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 6680.658500] Key type lgssc unregistered [ 6680.949964] LNet: 225926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6680.963943] LNetError: 225926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6680.988739] LNet: Removed LNI 192.168.203.142@tcp [ 6681.815251] Key type .llcrypt unregistered [ 6681.817981] Key type ._llcrypt unregistered [ 6707.383427] Key type ._llcrypt registered [ 6707.387072] Key type .llcrypt registered [ 6707.515479] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 6722.875852] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 6723.995920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6724.039598] alg: No test for adler32 (adler32-zlib) [ 6725.172395] Lustre: Lustre: Build Version: 2.17.58_3_g671df26 [ 6725.456213] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 6727.175263] Key type lgssc registered [ 6728.366977] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6762.193722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6762.214608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6763.489537] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6763.536966] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6763.672651] Lustre: lustre-MDT0000: new disk, initializing [ 6763.810470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6763.829317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6768.434513] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6778.740771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6778.864706] Lustre: 229280:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6778.930346] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6778.946543] Lustre: Skipped 1 previous similar message [ 6779.058391] Lustre: lustre-MDT0001: new disk, initializing [ 6779.151344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6779.208552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6779.216502] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6783.526356] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6792.455637] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6797.332814] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6803.328218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6803.678895] Lustre: lustre-OST0000: new disk, initializing [ 6803.682216] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6803.699439] Lustre: 231249:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6803.787525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6810.259816] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6810.683239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6810.689100] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6810.808249] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6819.123085] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6823.866555] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6824.133439] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6829.035026] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6829.319794] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6831.584482] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6831.590876] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6831.594475] LustreError: Skipped 1 previous similar message [ 6831.598215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6831.605721] Lustre: Skipped 1 previous similar message [ 6831.623471] Lustre: Skipped 1 previous similar message [ 6836.722054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6836.728790] Lustre: Skipped 1 previous similar message [ 6837.574523] Lustre: server umount lustre-OST0000 complete [ 6841.824554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6841.840616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6841.856422] Lustre: Skipped 1 previous similar message [ 6846.949136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6846.952672] Lustre: Skipped 1 previous similar message [ 6847.704131] Lustre: server umount lustre-MDT0000 complete [ 6850.662384] LustreError: 229274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788744648 with bad export cookie 15214412014477401474 [ 6850.665343] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6850.673552] LustreError: 229274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6851.105429] Lustre: server umount lustre-MDT0001 complete [ 6873.109485] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 6884.533295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6884.898386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6889.416879] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6889.957565] LustreError: 234184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6889.986424] LustreError: 234184:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6895.072950] LustreError: 234185:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6897.969048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6898.385925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6903.635288] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6906.665657] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6913.392714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6913.762303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6920.095412] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6928.455397] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6928.728079] Lustre: lustre-OST0001: new disk, initializing [ 6928.731273] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6928.740123] Lustre: 236489:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6930.765678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6930.777440] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6930.951064] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6935.387147] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 6946.549722] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6952.206887] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6963.938472] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 21:32:40 (1788744760) [ 6966.757269] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6966.758844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6966.794917] Lustre: Skipped 4 previous similar messages [ 6966.800224] Lustre: Skipped 2 previous similar messages [ 6972.251084] Lustre: server umount lustre-MDT0000 complete [ 6976.000899] LustreError: 234166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788744773 with bad export cookie 15214412014477402363 [ 6976.003954] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6976.013163] LustreError: 234166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6976.607712] Lustre: server umount lustre-MDT0001 complete [ 6989.734963] Lustre: server umount lustre-OST0000 complete [ 6992.863159] Lustre: 227504:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788744774/real 1788744774] req@ffff9e6c1fee3800 x1875634577056896/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788744790 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6992.875292] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6993.557740] Lustre: server umount lustre-OST0001 complete [ 7000.199480] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 7011.082501] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 7047.421828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7047.692714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7047.734596] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7047.815158] Lustre: lustre-MDT0000: new disk, initializing [ 7047.934835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7047.953760] Lustre: Skipped 1 previous similar message [ 7047.977272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7053.085531] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7064.111862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7064.237811] Lustre: 240433:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7064.303898] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7064.308308] Lustre: Skipped 1 previous similar message [ 7064.385534] Lustre: lustre-MDT0001: new disk, initializing [ 7064.568750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7064.583825] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7069.974660] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7079.828677] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7086.522846] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7093.731721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7093.997957] Lustre: lustre-OST0000: new disk, initializing [ 7094.003954] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7094.013584] Lustre: 242404:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7094.118964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7094.130943] Lustre: Skipped 1 previous similar message [ 7095.727576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7095.734945] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7095.898386] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7100.559697] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7110.266207] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7115.373606] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7115.583810] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7120.125491] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7120.327761] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7126.496714] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7126.497500] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7126.518186] LustreError: Skipped 1 previous similar message [ 7126.530709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7126.530719] Lustre: Skipped 5 previous similar messages [ 7126.565442] Lustre: Skipped 1 previous similar message [ 7128.674767] Lustre: server umount lustre-OST0000 complete [ 7136.227091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7136.237093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7138.655186] Lustre: server umount lustre-MDT0000 complete [ 7141.862206] LustreError: 241874:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7141.889410] LustreError: 241874:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7142.337260] LustreError: 240425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788744939 with bad export cookie 15214412014477404498 [ 7142.341071] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7142.358568] LustreError: 240425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7142.743885] Lustre: server umount lustre-MDT0001 complete [ 7150.588862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7150.974879] LustreError: 244617:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7155.683969] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7156.198677] LustreError: 244618:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7156.217970] LustreError: 244618:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7163.809687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7164.205916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7164.218869] Lustre: Skipped 1 previous similar message [ 7168.562882] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7175.728074] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7181.198185] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7187.298710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7194.116790] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7202.156713] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7349.734642] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7349.736916] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7349.752300] Lustre: Skipped 1 previous similar message [ 7349.758855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7349.761461] Lustre: Skipped 3 previous similar messages [ 7354.482905] Lustre: server umount lustre-OST0000 complete [ 7365.474420] Lustre: server umount lustre-MDT0000 complete [ 7369.857478] LustreError: 244600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788745167 with bad export cookie 15214412014477405380 [ 7369.872495] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7369.882017] LustreError: 244600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7370.256884] Lustre: server umount lustre-MDT0001 complete [ 7472.605334] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 7475.435208] Key type lgssc unregistered [ 7475.767427] LNet: 248224:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7475.779343] LNetError: 248224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7475.800186] LNet: Removed LNI 192.168.203.142@tcp [ 7476.677328] Key type .llcrypt unregistered [ 7476.679739] Key type ._llcrypt unregistered [ 7500.054834] Key type ._llcrypt registered [ 7500.057668] Key type .llcrypt registered [ 7500.213627] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 7516.097340] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 7516.996850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7517.023383] alg: No test for adler32 (adler32-zlib) [ 7518.224823] Lustre: Lustre: Build Version: 2.17.58_3_g671df26 [ 7518.533527] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 7520.232358] Key type lgssc registered [ 7521.329321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7564.195956] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 7575.752602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7575.811139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7577.205308] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7577.245741] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7577.371859] Lustre: lustre-MDT0000: new disk, initializing [ 7577.478731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7577.523290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7582.455785] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7592.667264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7592.845888] Lustre: 252292:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7592.957709] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7592.964809] Lustre: Skipped 1 previous similar message [ 7593.048197] Lustre: lustre-MDT0001: new disk, initializing [ 7593.184993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7593.232250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7593.246739] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7597.265815] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7601.681029] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7607.825312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7608.094330] Lustre: lustre-OST0000: new disk, initializing [ 7608.100682] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7608.106839] Lustre: 253924:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7608.236208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7613.533130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7613.539933] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7613.691587] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7614.038391] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7625.316271] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7625.501966] Lustre: lustre-OST0001: new disk, initializing [ 7625.510162] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7625.519199] Lustre: 254795:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7625.662657] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7632.070515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7632.082089] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7632.169078] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7633.342304] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7643.643551] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7647.903584] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7660.009547] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7660.020043] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7660.039696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7663.074490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7663.078142] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7663.086356] Lustre: Skipped 3 previous similar messages [ 7665.144923] Lustre: server umount lustre-MDT0000 complete [ 7670.073132] LustreError: 252284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788745467 with bad export cookie 2887881044681822231 [ 7670.075012] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7670.083372] LustreError: 252284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7670.316462] Lustre: server umount lustre-MDT0001 complete [ 7684.938693] Lustre: server umount lustre-OST0000 complete [ 7699.420671] Lustre: server umount lustre-OST0001 complete [ 7715.047902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7724.965089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7735.412083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7746.165074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7767.124918] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 7777.774251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7777.805872] Lustre: lustre-MDT0000: reset Object Index mappings [ 7778.283345] LustreError: 259810:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7778.304829] LustreError: 259810:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7778.373408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7783.397170] LustreError: 259811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7783.424090] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7787.509880] LustreError: 259810:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7792.613030] LustreError: 259811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7792.744141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7792.759745] Lustre: lustre-MDT0001: reset Object Index mappings [ 7793.259514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7797.875400] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7800.971694] Lustre: 260952:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7807.658068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7815.195149] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7818.403173] LustreError: 261305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7822.524484] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7824.219523] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7824.470068] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7824.474363] Lustre: Skipped 1 previous similar message [ 7829.501328] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 7831.403353] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7838.903960] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7842.441464] Lustre: 262823:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7849.459698] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7849.469178] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7849.502091] Lustre: Skipped 2 previous similar messages [ 7849.517361] LustreError: 259811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7849.544162] LustreError: 259811:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7849.666690] Lustre: server umount lustre-MDT0000 complete [ 7853.659327] LustreError: 259791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788745651 with bad export cookie 2887881044681829014 [ 7853.682582] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7854.217247] Lustre: server umount lustre-MDT0001 complete [ 7868.565921] Lustre: server umount lustre-OST0000 complete [ 7870.435331] Lustre: 249797:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788745652/real 1788745652] req@ffff9e6c34a22300 x1875635408593920/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788745668 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7870.470744] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7870.486826] Lustre: Skipped 3 previous similar messages [ 7872.097720] Lustre: server umount lustre-OST0001 complete [ 7889.661877] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 21:48:05 (1788745685) [ 7906.634445] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 7915.155532] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 7948.348776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7948.608734] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7948.634306] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7948.703367] Lustre: lustre-MDT0000: new disk, initializing [ 7948.790518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7948.817594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7953.356400] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7963.105683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7963.205753] Lustre: 267075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7963.281730] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7963.288076] Lustre: Skipped 1 previous similar message [ 7963.397084] Lustre: lustre-MDT0001: new disk, initializing [ 7963.579372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7963.592171] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7967.932321] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 7977.775446] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7983.369480] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7990.044600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7990.411089] Lustre: lustre-OST0000: new disk, initializing [ 7990.419723] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7990.429510] Lustre: 269044:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7992.417179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7992.428953] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7992.564911] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7996.532590] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8005.595566] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8010.038266] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8010.251538] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8014.274522] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8014.470704] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8018.400707] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8018.407720] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8018.413698] LustreError: Skipped 1 previous similar message [ 8018.416856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8018.440416] Lustre: Skipped 1 previous similar message [ 8022.494667] Lustre: server umount lustre-OST0000 complete [ 8028.640401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8028.643153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8028.664451] Lustre: Skipped 1 previous similar message [ 8028.682075] Lustre: Skipped 2 previous similar messages [ 8032.688657] Lustre: server umount lustre-MDT0000 complete [ 8033.760513] LustreError: 267082:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8033.773044] LustreError: 267082:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 8036.365325] LustreError: 267870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788745833 with bad export cookie 2887881044681831114 [ 8036.366505] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8036.375205] LustreError: 267870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8036.694453] Lustre: server umount lustre-MDT0001 complete [ 8056.500987] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 8066.408097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8066.828607] LustreError: 271983:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8066.899888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8066.909778] Lustre: Skipped 2 previous similar messages [ 8071.244916] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8078.567529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8082.494403] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8085.103630] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8092.282397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8098.209952] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8106.535263] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8106.700119] Lustre: lustre-OST0001: new disk, initializing [ 8106.704056] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8106.712512] Lustre: 274290:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8108.247779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8108.254591] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8108.397562] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8114.684787] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8125.726360] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8129.373433] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8135.175691] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 8135.624788] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 8150.499209] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8150.505117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8150.529270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8154.832502] Lustre: server umount lustre-MDT0000 complete [ 8157.670793] LustreError: 272711:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8157.708029] LustreError: 272711:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 8158.701909] LustreError: 271963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788745956 with bad export cookie 2887881044681831989 [ 8158.725410] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8159.124154] Lustre: server umount lustre-MDT0001 complete [ 8173.362591] Lustre: server umount lustre-OST0000 complete [ 8185.699602] Lustre: server umount lustre-OST0001 complete [ 8200.451111] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 8209.739978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8210.173571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8210.176735] Lustre: Skipped 3 previous similar messages [ 8214.437930] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8222.560173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8227.859132] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8230.493483] Lustre: 278775:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8236.243446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8241.817754] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8249.602753] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8255.649601] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8255.992757] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 8256.010169] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 8262.777134] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8266.088726] Lustre: 280648:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8270.850614] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 8270.905571] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 8282.838465] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 21:54:38 (1788746078) [ 8297.976049] Lustre: server umount lustre-MDT0000 complete [ 8300.004888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8300.022424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8300.042017] Lustre: Skipped 3 previous similar messages [ 8300.046663] LustreError: 278365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8300.077872] LustreError: 278365:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 8302.148542] LustreError: 278376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788746099 with bad export cookie 2887881044681838681 [ 8302.163937] LustreError: 278376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8302.167033] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8304.656841] Lustre: server umount lustre-MDT0001 complete [ 8322.527186] Lustre: 249797:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788746104/real 1788746104] req@ffff9e6c352b0a80 x1875635408780800/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788746120 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8324.664153] Lustre: server umount lustre-OST0000 complete [ 8327.647097] Lustre: 249798:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788746109/real 1788746109] req@ffff9e6c407dd180 x1875635408781184/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788746125 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8329.096382] Lustre: server umount lustre-OST0001 complete [ 8335.729761] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 8345.638491] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 8389.158301] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 8400.766591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8400.994368] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8401.016647] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8401.078773] Lustre: lustre-MDT0000: new disk, initializing [ 8401.175560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8405.979463] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8416.757174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8416.861301] Lustre: 285175:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8417.184842] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8423.051936] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8437.531255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8437.890510] Lustre: 286809:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8439.826551] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8444.734935] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8455.037041] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8455.194639] Lustre: 287681:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8456.483288] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8456.497219] Lustre: Skipped 1 previous similar message [ 8456.580980] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8461.691538] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8470.699329] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8473.937347] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8473.939985] Lustre: Skipped 1 previous similar message [ 8487.134892] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 21:58:03 (1788746283) [ 8491.342947] Lustre: server umount lustre-MDT0000 complete [ 8492.521790] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8492.539899] Lustre: Skipped 7 previous similar messages [ 8495.169730] LustreError: 285169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788746292 with bad export cookie 2887881044681846402 [ 8495.173632] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8495.198316] LustreError: 285169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8495.669732] Lustre: server umount lustre-MDT0001 complete [ 8510.229767] Lustre: server umount lustre-OST0000 complete [ 8513.504237] Lustre: 249800:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788746295/real 1788746295] req@ffff9e6c3e586680 x1875635408850688/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788746311 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8514.315455] Lustre: server umount lustre-OST0001 complete [ 8522.561479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8523.161674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8523.170483] Lustre: Skipped 7 previous similar messages [ 8528.380375] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8537.113548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8542.645685] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8550.889732] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8557.240406] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8566.867634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8568.823062] LustreError: 291752:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8568.868080] LustreError: 291752:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 8576.802118] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8585.459760] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8602.910676] loop1: detected capacity change from 0 to 204800000 [ 8603.109446] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8603.133190] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8603.156299] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8608.461421] loop1: detected capacity change from 0 to 204800000 [ 8613.679028] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8613.744679] LustreError: 289912:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs10300: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 8613.754808] LustreError: 289912:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs10300 [ 8613.759150] LustreError: 289912:0:(mgs_handler.c:517:mgs_target_reg()) fs10300-MDT0001: Error checking target index: rc = -13 [ 8613.770673] LustreError: fs10300-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 8613.790551] LustreError: 293641:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [ 8613.796245] LustreError: 293641:0:(tgt_mount.c:2132:server_put_super()) no obd fs10300-MDT0001 [ 8613.805751] LustreError: 293641:0:(tgt_mount.c:116:server_deregister_mount()) fs10300-MDT0001 not registered [ 8613.845989] Lustre: server umount fs10300-MDT0001 complete [ 8613.849271] LustreError: 293641:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 8621.521305] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8621.697401] Lustre: Setting parameter fs10300-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs10300-MDT0001 [ 8621.777078] Lustre: srv-fs10300-MDT0001: No data found on store. Initialize space. [ 8621.786446] Lustre: Skipped 4 previous similar messages [ 8621.863181] Lustre: fs10300-MDT0001: new disk, initializing [ 8621.865503] Lustre: Skipped 3 previous similar messages [ 8622.052917] Lustre: srv-fs10300-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8622.063504] Lustre: 294107:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs10300-MDT0001-mdtlov: can't create new llog: rc = -115 [ 8622.070379] LustreError: 294107:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs10300-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 8626.007825] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8633.730205] Lustre: server umount fs10300-MDT0001 complete [ 8649.553358] loop1: detected capacity change from 0 to 204800000 [ 8649.715068] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 8655.431179] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8655.914251] Lustre: 295675:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs10300-MDT0001-mdtlov: can't create new llog: rc = -115 [ 8655.918817] LustreError: 295675:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs10300-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 8661.496279] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8667.106791] Lustre: server umount fs10300-MDT0001 complete [ 8680.421107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8680.445361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8680.486732] Lustre: Skipped 3 previous similar messages [ 8680.513538] Lustre: Skipped 6 previous similar messages [ 8694.239319] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8694.511526] Lustre: server umount lustre-MDT0000 complete [ 8703.488070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8703.710401] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8708.899267] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8709.131666] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8716.957544] loop1: detected capacity change from 0 to 204800000 [ 8722.954606] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8723.348465] Lustre: 298137:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs10300-MDT0001-mdtlov: can't create new llog: rc = -115 [ 8723.357418] LustreError: 298137:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs10300-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 8728.336689] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8733.338624] Lustre: server umount fs10300-MDT0001 complete [ 8757.216259] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8757.223713] LustreError: Skipped 1 previous similar message [ 8757.243477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8757.250574] Lustre: Skipped 9 previous similar messages [ 8762.228946] Lustre: server umount lustre-OST0000 complete [ 8772.546493] Lustre: server umount lustre-MDT0000 complete [ 8775.780715] LustreError: 289902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788746573 with bad export cookie 2887881044681849083 [ 8775.793358] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8775.795677] LustreError: 289902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8776.329274] Lustre: server umount lustre-MDT0001 complete [ 8787.687702] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 8790.865524] Key type lgssc unregistered [ 8791.182814] LNet: 300482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8791.200306] LNetError: 300482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8791.213792] LNet: Removed LNI 192.168.203.142@tcp [ 8792.112223] Key type .llcrypt unregistered [ 8792.114419] Key type ._llcrypt unregistered [ 8812.712209] Key type ._llcrypt registered [ 8812.714167] Key type .llcrypt registered [ 8812.894722] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 8814.300489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8814.430185] alg: No test for adler32 (adler32-zlib) [ 8815.512860] Lustre: Lustre: Build Version: 2.17.58_3_g671df26 [ 8815.725252] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 8817.415183] Key type lgssc registered [ 8818.556652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8826.680422] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8826.708527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8828.262677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8833.037236] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8837.605803] LustreError: 301751:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8837.618529] LustreError: 301751:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8841.968676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8842.442910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8847.072173] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8854.856210] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8860.884785] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8867.617942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8868.111276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8873.261926] LustreError: 303580:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8874.537514] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8878.577440] LustreError: 303581:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8882.173736] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8883.686920] LustreError: 303882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8890.338323] LustreError: 303580:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8890.389582] LustreError: 303580:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8900.573515] LustreError: 303882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8900.597981] LustreError: 303882:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 8903.961088] loop1: detected capacity change from 0 to 204800000 [ 8904.255354] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8904.275254] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8904.293853] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8909.611529] loop1: detected capacity change from 0 to 204800000 [ 8915.362730] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8915.603418] Lustre: fs13970-OST0000: new disk, initializing [ 8915.609353] Lustre: srv-fs13970-OST0000: No data found on store. Initialize space. [ 8915.613775] Lustre: 305484:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8915.724984] Lustre: fs13970-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8917.485119] LustreError: 303580:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8917.499746] LustreError: 303580:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 8922.486903] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8931.121629] Lustre: server umount fs13970-OST0000 complete [ 8943.423323] loop1: detected capacity change from 0 to 204800000 [ 8949.099628] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8949.376123] Lustre: fs13970-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8951.784115] LustreError: 303581:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8951.817238] LustreError: 303581:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [ 8953.334317] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 8956.069315] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 8965.317912] loop2: detected capacity change from 0 to 204800000 [ 8965.679470] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8965.696665] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8965.711180] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8968.681030] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 8970.517611] loop2: detected capacity change from 0 to 204800000 [ 8975.792763] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8975.880953] LustreError: fs13970-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 8975.904227] LustreError: fs13970-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 8975.914162] LustreError: 308522:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [ 8975.921322] LustreError: 308522:0:(tgt_mount.c:2132:server_put_super()) no obd fs13970-OST0001 [ 8975.933342] LustreError: 308522:0:(tgt_mount.c:116:server_deregister_mount()) fs13970-OST0001 not registered [ 8975.965281] Lustre: server umount fs13970-OST0001 complete [ 8975.967448] LustreError: 308522:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 8986.011724] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8986.226792] Lustre: fs13970-OST0001: new disk, initializing [ 8986.236737] Lustre: srv-fs13970-OST0001: No data found on store. Initialize space. [ 8986.244502] Lustre: 309074:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8986.346768] Lustre: fs13970-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8993.208501] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9000.959875] Lustre: server umount fs13970-OST0000 complete [ 9010.526465] Lustre: server umount fs13970-OST0001 complete [ 9018.330818] LustreError: 303580:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9018.358867] LustreError: 303580:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [ 9022.386543] loop1: detected capacity change from 0 to 204800000 [ 9027.733299] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9028.046289] Lustre: fs13970-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9034.101707] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9039.938699] loop2: detected capacity change from 0 to 204800000 [ 9044.946707] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9051.179993] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9060.097912] Lustre: server umount fs13970-OST0000 complete [ 9069.389738] Lustre: server umount fs13970-OST0001 complete [ 9083.361174] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9083.377491] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9083.418786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9086.432065] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9086.440193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9086.444064] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9086.470625] Lustre: Skipped 1 previous similar message [ 9091.558230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9096.159953] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9096.440386] Lustre: server umount lustre-OST0000 complete [ 9101.792519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9101.794165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9101.824251] Lustre: Skipped 1 previous similar message [ 9101.835802] Lustre: Skipped 2 previous similar messages [ 9106.564669] Lustre: server umount lustre-MDT0000 complete [ 9109.907356] LustreError: 301732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788746907 with bad export cookie 8128447410289469268 [ 9109.935475] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9110.447584] Lustre: server umount lustre-MDT0001 complete [ 9121.640228] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 9125.024583] Key type lgssc unregistered [ 9125.359459] LNet: 314998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9125.376562] LNetError: 314998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9125.393697] LNet: Removed LNI 192.168.203.142@tcp [ 9126.535181] Key type .llcrypt unregistered [ 9126.540539] Key type ._llcrypt unregistered [ 9156.514276] Key type ._llcrypt registered [ 9156.521881] Key type .llcrypt registered [ 9156.673598] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 9171.401647] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 9172.601543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9172.767153] alg: No test for adler32 (adler32-zlib) [ 9173.896537] Lustre: Lustre: Build Version: 2.17.58_3_g671df26 [ 9174.176407] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 9175.879143] Key type lgssc registered [ 9177.067560] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9207.893989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9207.929518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9209.231234] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9209.291354] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9209.392905] Lustre: lustre-MDT0000: new disk, initializing [ 9209.508616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9209.525484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9214.030331] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9223.343791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9223.433799] Lustre: 318851:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9223.477568] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9223.480692] Lustre: Skipped 1 previous similar message [ 9223.537783] Lustre: lustre-MDT0001: new disk, initializing [ 9223.607824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9223.637051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9223.652226] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9228.373748] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9237.519183] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9242.400277] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9248.237471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9248.466709] Lustre: lustre-OST0000: new disk, initializing [ 9248.473865] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9248.479649] Lustre: 320819:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9248.583849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9254.547750] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9257.025318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9257.033381] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9257.132725] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9264.293861] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9266.319626] Lustre: Mounted lustre-client - version 2.17.58_3_g671df26 [ 9267.333272] Lustre: Unmounted lustre-client [ 9276.904495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9276.925318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9276.935828] Lustre: Skipped 1 previous similar message [ 9281.890423] Lustre: server umount lustre-OST0000 complete [ 9287.142551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9287.144365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9287.154969] Lustre: Skipped 2 previous similar messages [ 9291.678780] Lustre: server umount lustre-MDT0000 complete [ 9294.901961] LustreError: 318844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788747092 with bad export cookie 2513277164791303705 [ 9294.915023] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9294.923713] LustreError: 318844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9295.186291] Lustre: server umount lustre-MDT0001 complete [ 9304.853638] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 9307.385622] Key type lgssc unregistered [ 9317.535399] LNet: 1 local NIs in recovery (showing 1): 192.168.203.142@tcp [ 9317.855996] LNet: 322980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9317.863178] LNetError: 322980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9397.734773] LNet: Removed LNI 192.168.203.142@tcp [ 9398.391242] Key type .llcrypt unregistered [ 9398.394298] Key type ._llcrypt unregistered [ 9421.703704] Key type ._llcrypt registered [ 9421.711974] Key type .llcrypt registered [ 9422.043765] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 9447.465899] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 9449.305493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9449.409540] alg: No test for adler32 (adler32-zlib) [ 9451.026822] Lustre: Lustre: Build Version: 2.17.58_3_g671df26 [ 9451.301168] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 9453.039166] Key type lgssc registered [ 9454.463902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9489.790418] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 22:14:45 (1788747285) [ 9496.850967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9496.892874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9498.210816] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9498.250990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9498.327184] Lustre: lustre-MDT0000: new disk, initializing [ 9498.401350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9498.427561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9503.055809] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9514.223168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9514.436299] Lustre: 326221:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9514.493086] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9514.498531] Lustre: Skipped 1 previous similar message [ 9514.662614] Lustre: lustre-MDT0001: new disk, initializing [ 9514.807093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9514.865670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9514.880963] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9519.775858] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9529.081508] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9534.678975] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9540.752571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9541.017374] Lustre: lustre-OST0000: new disk, initializing [ 9541.024968] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9541.034761] Lustre: 328189:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9541.141483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9546.644352] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9549.931471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9549.947858] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9550.088589] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9558.023913] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9575.904566] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9575.915964] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9575.928827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9575.938190] Lustre: Skipped 1 previous similar message [ 9578.033637] Lustre: server umount lustre-OST0000 complete [ 9581.025803] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9581.045856] Lustre: Skipped 1 previous similar message [ 9581.054450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9581.059630] Lustre: Skipped 1 previous similar message [ 9586.147364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9586.166146] Lustre: Skipped 1 previous similar message [ 9591.264258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9591.275490] Lustre: Skipped 1 previous similar message [ 9595.359918] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9595.619687] Lustre: server umount lustre-MDT0000 complete [ 9599.196095] LustreError: 326214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788747396 with bad export cookie 13380580950108378477 [ 9599.200026] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9599.207279] LustreError: 326214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9599.473747] Lustre: server umount lustre-MDT0001 complete [ 9609.543290] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 9612.648275] Key type lgssc unregistered [ 9612.960664] LNet: 330095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9612.965563] LNetError: 330095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9613.991899] LNet: Removed LNI 192.168.203.142@tcp [ 9614.927701] Key type .llcrypt unregistered [ 9614.933975] Key type ._llcrypt unregistered [ 9634.646211] Key type ._llcrypt registered [ 9634.648279] Key type .llcrypt registered [ 9634.747508] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 9635.792195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9635.841422] alg: No test for adler32 (adler32-zlib) [ 9636.895655] Lustre: Lustre: Build Version: 2.17.58_3_g671df26 [ 9637.111242] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 9638.783273] Key type lgssc registered [ 9639.801268] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9647.150836] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9647.183204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9648.778257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9653.370113] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9659.368141] LustreError: 331364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9659.382122] LustreError: 331364:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9660.814212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9661.091717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9665.058976] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9671.990859] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9676.714202] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9682.874458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9683.264340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9689.719238] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9697.228650] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9706.795270] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9708.238545] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9712.608477] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9712.613666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9712.622772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9714.155101] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9714.167422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9715.188020] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.42@tcp (stopping) [ 9715.191149] Lustre: Skipped 2 previous similar messages [ 9717.947658] Lustre: server umount lustre-MDT0000 complete [ 9719.266274] LustreError: 332937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9719.298299] LustreError: 332937:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9720.288056] LustreError: 331360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9721.067258] LustreError: 332098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788747518 with bad export cookie 13538439487668996533 [ 9721.075171] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9721.078207] LustreError: 332098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9724.387127] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9724.392346] LustreError: 332937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9724.400437] Lustre: Skipped 1 previous similar message [ 9724.401833] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9727.507619] Lustre: server umount lustre-MDT0001 complete [ 9731.140530] Lustre: server umount lustre-OST0000 complete [ 9739.109705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9744.739416] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9748.092691] Lustre: 335544:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.142@tcp [ 9750.404860] Lustre: server umount MGS complete [ 9760.547502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9761.409905] LustreError: 336116:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9761.448931] LustreError: 336116:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9761.541327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9766.428320] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9771.490724] LustreError: 336112:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9771.514236] LustreError: 336112:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9774.857171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9775.375919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9780.156904] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9787.752456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9788.118987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9793.836910] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9801.070277] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9802.577518] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 9807.348606] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9808.795691] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9811.936118] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9811.939305] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9811.941711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9811.941718] Lustre: Skipped 2 previous similar messages [ 9811.961243] Lustre: Skipped 1 previous similar message [ 9814.152599] Lustre: server umount lustre-OST0000 complete [ 9821.666618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9821.669928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9821.712336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9821.719701] Lustre: Skipped 1 previous similar message [ 9824.342957] Lustre: server umount lustre-MDT0000 complete [ 9827.299473] LustreError: 336116:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9827.323541] LustreError: 336116:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9828.352992] LustreError: 337573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788747625 with bad export cookie 13538439487668997373 [ 9828.359400] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9828.377666] LustreError: 337573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9828.798090] Lustre: server umount lustre-MDT0001 complete [ 9840.580304] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 9843.361416] Key type lgssc unregistered [ 9843.792350] LNet: 339551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9843.804335] LNetError: 339551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9843.861836] LNet: Removed LNI 192.168.203.142@tcp [ 9844.829373] Key type .llcrypt unregistered [ 9844.831813] Key type ._llcrypt unregistered [ 9860.948344] Key type ._llcrypt registered [ 9860.951205] Key type .llcrypt registered [ 9861.071537] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 9876.387424] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 9877.337100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9877.358873] alg: No test for adler32 (adler32-zlib) [ 9878.476961] Lustre: Lustre: Build Version: 2.17.58_3_g671df26 [ 9878.777667] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 9880.487462] Key type lgssc registered [ 9881.652564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9916.945993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9916.965968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9918.160981] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9918.192860] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9918.249923] Lustre: lustre-MDT0000: new disk, initializing [ 9918.341550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9918.371118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9922.726721] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9933.871271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9934.009825] Lustre: 342708:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9934.118571] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9934.132069] Lustre: Skipped 1 previous similar message [ 9934.184580] Lustre: lustre-MDT0001: new disk, initializing [ 9934.286451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9934.334761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9934.353963] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9938.860980] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9950.692612] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9958.456320] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9965.673033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9965.957629] Lustre: lustre-OST0000: new disk, initializing [ 9965.974087] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9966.001556] Lustre: 344680:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9966.096486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9973.006440] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 9974.860091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9974.871908] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9975.013800] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9980.917931] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9985.239144] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9985.546217] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9991.486978] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9991.781193] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9995.748042] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9995.749316] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9995.752867] LustreError: Skipped 1 previous similar message [ 9995.754652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9995.780982] Lustre: Skipped 1 previous similar message [10000.479521] Lustre: server umount lustre-OST0000 complete [10005.985828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10005.987815] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10006.010531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10006.025485] Lustre: Skipped 1 previous similar message [10006.043787] Lustre: Skipped 2 previous similar messages [10011.109537] Lustre: server umount lustre-MDT0000 complete [10015.779365] LustreError: 342701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788747813 with bad export cookie 5619514698166836550 [10015.779944] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10015.785162] LustreError: 342701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10016.086547] Lustre: server umount lustre-MDT0001 complete [10024.667315] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 22:23:40 (1788747820) [10037.709810] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [10046.697544] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [10084.441743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10084.787262] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10084.868426] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10084.962941] Lustre: lustre-MDT0000: new disk, initializing [10085.053513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10085.068882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10089.315077] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [10098.704189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10098.800314] Lustre: 349436:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10098.848948] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10098.852651] Lustre: Skipped 1 previous similar message [10098.933119] Lustre: lustre-MDT0001: new disk, initializing [10099.057416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10099.104145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10099.114081] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10103.449651] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [10112.057380] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10117.138049] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10123.953383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10124.127214] Lustre: lustre-OST0000: new disk, initializing [10124.130561] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10124.135029] Lustre: 351405:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10124.210214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10130.360409] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [10134.071189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10134.083932] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10134.160201] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10137.772717] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10170.341458] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10170.344862] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10170.386778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10175.459863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10175.474593] Lustre: Skipped 2 previous similar messages [10175.910592] Lustre: server umount lustre-OST0000 complete [10180.576536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10180.587607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10180.600206] Lustre: Skipped 2 previous similar messages [10185.944811] Lustre: server umount lustre-MDT0000 complete [10189.024595] LustreError: 349430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788747986 with bad export cookie 5619514698166837432 [10189.025802] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10189.034344] LustreError: 349430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10189.337354] Lustre: server umount lustre-MDT0001 complete [10198.672195] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [10201.648793] Key type lgssc unregistered [10202.090876] LNet: 353578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10202.097265] LNetError: 353578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10203.177439] LNet: Removed LNI 192.168.203.142@tcp [10203.880581] Key type .llcrypt unregistered [10203.887134] Key type ._llcrypt unregistered [10282.597195] Key type ._llcrypt registered [10282.599098] Key type .llcrypt registered [10282.687937] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [10282.930068] Key type .llcrypt unregistered [10282.933091] Key type ._llcrypt unregistered [10371.139638] Key type ._llcrypt registered [10371.141277] Key type .llcrypt registered [10371.297945] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [10371.803316] Key type .llcrypt unregistered [10371.805080] Key type ._llcrypt unregistered [10400.866135] Key type ._llcrypt registered [10400.868298] Key type .llcrypt registered [10400.923593] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [10401.111786] Key type .llcrypt unregistered [10401.113714] Key type ._llcrypt unregistered [10431.434858] Key type ._llcrypt registered [10431.437048] Key type .llcrypt registered [10431.554871] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [10431.850394] Key type .llcrypt unregistered [10431.855828] Key type ._llcrypt unregistered [10501.681766] Key type ._llcrypt registered [10501.684436] Key type .llcrypt registered [10501.774594] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [10502.109572] Key type .llcrypt unregistered [10502.111465] Key type ._llcrypt unregistered [10530.644201] Key type ._llcrypt registered [10530.645907] Key type .llcrypt registered [10530.778505] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [10531.126963] Key type .llcrypt unregistered [10531.129982] Key type ._llcrypt unregistered [10560.793858] Key type ._llcrypt registered [10560.796596] Key type .llcrypt registered [10560.944173] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [10561.197451] Key type .llcrypt unregistered [10561.200524] Key type ._llcrypt unregistered [10586.643497] Key type ._llcrypt registered [10586.646301] Key type .llcrypt registered [10586.761465] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [10587.061288] Key type .llcrypt unregistered [10587.062940] Key type ._llcrypt unregistered [10617.895771] Key type ._llcrypt registered [10617.897639] Key type .llcrypt registered [10617.971480] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [10618.178818] Key type .llcrypt unregistered [10618.183624] Key type ._llcrypt unregistered [10650.080737] Key type ._llcrypt registered [10650.085631] Key type .llcrypt registered [10650.234852] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [10650.565092] Key type .llcrypt unregistered [10650.569619] Key type ._llcrypt unregistered [10716.545224] Key type ._llcrypt registered [10716.547579] Key type .llcrypt registered [10716.646770] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [10716.976694] Key type .llcrypt unregistered [10716.983124] Key type ._llcrypt unregistered [10746.003674] Key type ._llcrypt registered [10746.008294] Key type .llcrypt registered [10746.104935] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [10746.362428] Key type .llcrypt unregistered [10746.366400] Key type ._llcrypt unregistered [10774.523345] Key type ._llcrypt registered [10774.525096] Key type .llcrypt registered [10774.621316] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [10774.998936] Key type .llcrypt unregistered [10775.007714] Key type ._llcrypt unregistered [10821.938858] Key type ._llcrypt registered [10821.943055] Key type .llcrypt registered [10822.062936] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [10836.194428] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [10837.183771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10837.247331] alg: No test for adler32 (adler32-zlib) [10838.219751] Lustre: Lustre: Build Version: 2.17.58_3_g671df26 [10838.505498] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [10840.168152] Key type lgssc registered [10841.151119] Lustre: Echo OBD driver; http://www.lustre.org/ [10873.220576] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10873.237963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10874.488115] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [10874.531800] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [10874.612599] Lustre: tst250fs-MDT0000: new disk, initializing [10874.726863] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10874.749627] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10878.458930] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [10887.916260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10888.057342] Lustre: 373804:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10888.157509] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [10888.165709] Lustre: Skipped 1 previous similar message [10888.294582] Lustre: tst250fs-MDT0001: new disk, initializing [10888.436766] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10888.503570] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10888.519176] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10893.321971] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [10898.137708] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10904.397462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10904.717366] Lustre: tst250fs-OST0000: new disk, initializing [10904.723795] Lustre: tst250fs-OST0000: Not available for connect from 0@lo (not set up) [10904.729053] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [10904.740096] Lustre: 375442:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10904.870164] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10910.253370] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10910.276498] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10910.309528] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [10911.206858] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [10920.775534] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10920.981836] Lustre: tst250fs-OST0001: new disk, initializing [10920.989877] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [10920.993827] Lustre: tst250fs-OST0001: Not available for connect from 0@lo (not set up) [10921.008787] Lustre: 376316:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10921.125533] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10926.591770] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10926.614741] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10926.667868] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10927.366243] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [10932.595330] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [10935.376049] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [10935.386023] Lustre: Skipped 2 previous similar messages [10955.232293] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10955.244626] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10955.273110] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [10957.281437] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [10957.282917] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10957.310255] Lustre: Skipped 2 previous similar messages [10959.239773] Lustre: server umount tst250fs-MDT0000 complete [10962.415161] LustreError: 373817:0:(ldlm_lib.c:1190:target_handle_connect()) tst250fs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10962.444893] LustreError: 373817:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10962.874569] LustreError: 373798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788748760 with bad export cookie 7083781277509749415 [10962.892186] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10962.909165] LustreError: 373798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10963.235150] Lustre: server umount tst250fs-MDT0001 complete [10976.963187] Lustre: server umount tst250fs-OST0000 complete [10991.266941] Lustre: server umount tst250fs-OST0001 complete [11009.058891] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [11016.534296] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [11043.893248] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 22:40:40 (1788748840) [11045.323764] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [11059.242491] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [11068.457348] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [11095.842958] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10977 sec ========= 22:41:32 (1788748892) [11097.574101] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:41:33 (1788748893) === [11100.770894] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:41:36 (1788748896) === [11132.837408] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [11135.180804] Key type lgssc unregistered [11135.455202] LNet: 385450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11135.468204] LNetError: 385450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11135.488818] LNet: Removed LNI 192.168.203.142@tcp [11136.275741] Key type .llcrypt unregistered [11136.279187] Key type ._llcrypt unregistered