[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 506087962 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2640MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524576K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.003361] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005019] kvm-guest: setup PV IPIs [ 0.008857] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.011199] LSM: Security Framework initializing [ 0.012053] Yama: becoming mindful. [ 0.013042] SELinux: Initializing. [ 0.014072] *** VALIDATE selinux *** [ 0.023574] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028561] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029157] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030109] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032091] *** VALIDATE tmpfs *** [ 0.033384] *** VALIDATE proc *** [ 0.034266] *** VALIDATE cgroup *** [ 0.035013] *** VALIDATE cgroup2 *** [ 0.036302] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038148] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040053] Spectre V2 : User space: Vulnerable [ 0.041011] Speculative Store Bypass: Vulnerable [ 0.044546] debug: unmapping init [mem 0xffffffff84059000-0xffffffff84060fff] [ 0.046259] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047708] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048028] ... version: 2 [ 0.049017] ... bit width: 48 [ 0.050017] ... generic registers: 4 [ 0.051017] ... value mask: 0000ffffffffffff [ 0.052017] ... max period: 00007fffffffffff [ 0.053016] ... fixed-purpose events: 3 [ 0.054014] ... event mask: 000000070000000f [ 0.055280] rcu: Hierarchical SRCU implementation. [ 0.057482] smp: Bringing up secondary CPUs ... [ 0.058649] x86: Booting SMP configuration: [ 0.059029] .... node #0, CPUs: #1 #2 #3 [ 0.062342] smp: Brought up 1 node, 4 CPUs [ 0.064014] smpboot: Max logical packages: 1 [ 0.065021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.146579] node 0 deferred pages initialised in 78ms [ 0.151012] devtmpfs: initialized [ 0.152243] x86/mm: Memory block size: 128MB [ 0.154739] gcov: version magic: 0x41383552 [ 0.156299] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.159105] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.162395] pinctrl core: initialized pinctrl subsystem [ 0.164227] [ 0.164744] ************************************************************* [ 0.167014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169014] ** ** [ 0.172020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.174012] ** ** [ 0.176016] ** This means that this kernel is built to expose internal ** [ 0.179014] ** IOMMU data structures, which may compromise security on ** [ 0.181012] ** your system. ** [ 0.183015] ** ** [ 0.186013] ** If you see this message and you are not debugging the ** [ 0.188014] ** kernel, report this immediately to your vendor! ** [ 0.190025] ** ** [ 0.192034] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.195013] ************************************************************* [ 0.197772] NET: Registered protocol family 16 [ 0.199512] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.202058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.203049] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.207027] cpuidle: using governor menu [ 0.208713] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.210322] PCI: Using configuration type 1 for base access [ 0.211126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.219072] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.220020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.221246] cryptd: max_cpu_qlen set to 1000 [ 0.223223] ACPI: Added _OSI(Module Device) [ 0.224009] ACPI: Added _OSI(Processor Device) [ 0.224901] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.225028] ACPI: Added _OSI(Processor Aggregator Device) [ 0.228404] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.233367] ACPI: Interpreter enabled [ 0.234005] ACPI: PM: (supports S0 S3 S4 S5) [ 0.235009] ACPI: Using IOAPIC for interrupt routing [ 0.236090] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.238256] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.249487] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.253054] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.255018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.258094] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.263541] acpiphp: Slot [2] registered [ 0.265119] acpiphp: Slot [5] registered [ 0.267148] acpiphp: Slot [6] registered [ 0.268120] acpiphp: Slot [7] registered [ 0.269115] acpiphp: Slot [8] registered [ 0.271141] acpiphp: Slot [9] registered [ 0.272106] acpiphp: Slot [10] registered [ 0.274099] acpiphp: Slot [3] registered [ 0.275114] acpiphp: Slot [4] registered [ 0.277092] acpiphp: Slot [11] registered [ 0.278091] acpiphp: Slot [12] registered [ 0.280113] acpiphp: Slot [13] registered [ 0.281162] acpiphp: Slot [14] registered [ 0.283058] acpiphp: Slot [15] registered [ 0.284042] acpiphp: Slot [16] registered [ 0.284955] acpiphp: Slot [17] registered [ 0.286063] acpiphp: Slot [18] registered [ 0.287130] acpiphp: Slot [19] registered [ 0.288101] acpiphp: Slot [20] registered [ 0.290103] acpiphp: Slot [21] registered [ 0.291128] acpiphp: Slot [22] registered [ 0.293115] acpiphp: Slot [23] registered [ 0.294115] acpiphp: Slot [24] registered [ 0.296100] acpiphp: Slot [25] registered [ 0.297129] acpiphp: Slot [26] registered [ 0.298094] acpiphp: Slot [27] registered [ 0.300117] acpiphp: Slot [28] registered [ 0.301105] acpiphp: Slot [29] registered [ 0.303105] acpiphp: Slot [30] registered [ 0.304148] acpiphp: Slot [31] registered [ 0.305060] PCI host bridge to bus 0000:00 [ 0.307040] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.309030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.311029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.314025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.317028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.320026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.322229] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.325012] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.328202] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.338028] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.343520] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.346021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.348018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.351019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.353847] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.356877] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.359051] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.362842] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.368017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.382024] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.387852] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.394085] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.411019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.424017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.445021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.457060] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.466018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.473017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.488017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.496410] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.502017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.507018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.527015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.533873] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.542018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.549019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.564018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.574379] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.580014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.585014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.609026] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.621981] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.627015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.643016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.662022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.671259] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.674283] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.676321] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.677294] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.679131] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.683708] iommu: Default domain type: Passthrough [ 0.684430] SCSI subsystem initialized [ 0.685104] ACPI: bus type USB registered [ 0.686083] usbcore: registered new interface driver usbfs [ 0.688052] usbcore: registered new interface driver hub [ 0.689054] usbcore: registered new device driver usb [ 0.690118] pps_core: LinuxPPS API ver. 1 registered [ 0.691009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.693044] PTP clock support registered [ 0.695079] EDAC MC: Ver: 3.0.0 [ 0.696131] PCI: Using ACPI for IRQ routing [ 0.697603] NetLabel: Initializing [ 0.698007] NetLabel: domain hash size = 128 [ 0.699017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.700074] NetLabel: unlabeled traffic allowed by default [ 0.701149] vgaarb: loaded [ 0.703208] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.704008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.713623] clocksource: Switched to clocksource kvm-clock [ 0.810349] VFS: Disk quotas dquot_6.6.0 [ 0.812078] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.814956] *** VALIDATE ramfs *** [ 0.816271] *** VALIDATE hugetlbfs *** [ 0.818224] pnp: PnP ACPI init [ 0.820744] pnp: PnP ACPI: found 6 devices [ 0.842749] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.846323] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.848627] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.850914] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.853052] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.856407] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.859284] NET: Registered protocol family 2 [ 0.860934] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.864220] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.866411] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.869987] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.872379] TCP: Hash tables configured (established 65536 bind 65536) [ 0.874082] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.875855] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.877628] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.879422] NET: Registered protocol family 1 [ 0.881638] RPC: Registered named UNIX socket transport module. [ 0.882885] RPC: Registered udp transport module. [ 0.884161] RPC: Registered tcp transport module. [ 0.885158] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.886639] NET: Registered protocol family 44 [ 0.888325] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.890485] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.892657] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.895037] PCI: CLS 0 bytes, default 64 [ 0.896832] Unpacking initramfs... [ 2.244310] debug: unmapping init [mem 0xffff8c32bcc54000-0xffff8c32bffbffff] [ 2.247503] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.249907] software IO TLB: mapped [mem 0x00000000a1000000-0x00000000a5000000] (64MB) [ 2.252053] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.765899] Initialise system trusted keyrings [ 2.767744] Key type blacklist registered [ 2.769780] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.778681] zbud: loaded [ 2.781743] *** VALIDATE nfs *** [ 2.782949] *** VALIDATE nfs4 *** [ 2.784646] pstore: using deflate compression [ 2.788608] Platform Keyring initialized [ 2.884665] NET: Registered protocol family 38 [ 2.886990] Key type asymmetric registered [ 2.888732] Asymmetric key parser 'x509' registered [ 2.890716] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.893984] io scheduler mq-deadline registered [ 2.895808] io scheduler kyber registered [ 2.897523] io scheduler bfq registered [ 2.899595] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.901941] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.904600] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.906651] ACPI: Power Button [PWRF] [ 2.911436] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.916425] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.927299] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.938293] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.957456] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.986229] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.015194] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.021958] Non-volatile memory driver v1.3 [ 3.023626] Linux agpgart interface v0.103 [ 3.061077] virtio_blk virtio1: [vda] 134784 512-byte logical blocks (69.0 MB/65.8 MiB) [ 3.064036] vda: detected capacity change from 0 to 69009408 [ 3.084898] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.088405] vdb: detected capacity change from 0 to 1073741824 [ 3.101581] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.105693] vdc: detected capacity change from 0 to 2621440000 [ 3.119941] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.124109] vdd: detected capacity change from 0 to 2621440000 [ 3.139480] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.142720] vde: detected capacity change from 0 to 4294967296 [ 3.167893] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.170589] vdf: detected capacity change from 0 to 4294967296 [ 3.184197] libphy: Fixed MDIO Bus: probed [ 3.205925] usbcore: registered new interface driver usbserial_generic [ 3.209652] usbserial: USB Serial support registered for generic [ 3.212633] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.218579] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.222559] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.225457] mousedev: PS/2 mouse device common for all mice [ 3.228494] rtc_cmos 00:05: RTC can wake from S4 [ 3.231853] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.232205] rtc_cmos 00:05: registered as rtc0 [ 3.238707] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.238964] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.241616] intel_pstate: CPU model not supported [ 3.248274] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.251265] hid: raw HID events driver (C) Jiri Kosina [ 3.254959] usbcore: registered new interface driver usbhid [ 3.256984] usbhid: USB HID core driver [ 3.258442] drop_monitor: Initializing network drop monitor service [ 3.260927] Initializing XFRM netlink socket [ 3.263209] NET: Registered protocol family 10 [ 3.266446] Segment Routing with IPv6 [ 3.268233] NET: Registered protocol family 17 [ 3.270740] mpls_gso: MPLS GSO support [ 3.275992] RAS: Correctable Errors collector initialized. [ 3.278841] AVX version of gcm_enc/dec engaged. [ 3.280738] AES CTR mode by8 optimization enabled [ 3.364328] sched_clock: Marking stable (3364304610, 0)->(4288291773, -923987163) [ 3.368290] registered taskstats version 1 [ 3.371495] Loading compiled-in X.509 certificates [ 3.373384] zswap: loaded using pool lzo/zbud [ 3.398988] Key type big_key registered [ 3.413971] Key type encrypted registered [ 3.415576] ima: No TPM chip found, activating TPM-bypass! [ 3.417305] ima: Allocated hash algorithm: sha1 [ 3.419247] ima: No architecture policies found [ 3.421341] evm: Initialising EVM extended attributes: [ 3.423489] evm: security.selinux [ 3.424806] evm: security.ima [ 3.426228] evm: security.capability [ 3.427649] evm: HMAC attrs: 0x1 [ 3.430603] rtc_cmos 00:05: setting system clock to 2026-05-16 09:12:45 UTC (1778922765) [ 3.437562] debug: unmapping init [mem 0xffffffff85003000-0xffffffff851fffff] [ 3.441064] debug: unmapping init [mem 0xffffffff83d82000-0xffffffff84058fff] [ 3.451107] Write protecting the kernel read-only data: 28672k [ 3.455277] debug: unmapping init [mem 0xffffffff82403000-0xffffffff825fffff] [ 3.458455] debug: unmapping init [mem 0xffffffff82d14000-0xffffffff82dfffff] [ 3.498474] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.508637] systemd[1]: Detected virtualization kvm. [ 3.510682] systemd[1]: Detected architecture x86-64. [ 3.512860] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.540601] systemd[1]: No hostname configured. [ 3.542505] systemd[1]: Set hostname to . [ 3.544669] random: systemd: uninitialized urandom read (16 bytes read) [ 3.547132] systemd[1]: Initializing machine ID from random generator. [ 3.590537] random: ln: uninitialized urandom read (6 bytes read) [ 3.686105] random: systemd: uninitialized urandom read (16 bytes read) [ 3.689414] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.695893] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.700072] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.280363] device-mapper: uevent: version 1.0.3 [ 4.282136] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.889827] virtio_net virtio0 ens2: renamed from eth0 [ 4.923253] random: fast init done [ 5.024471] scsi host0: ata_piix [ 5.074436] scsi host1: ata_piix [ 5.076161] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.078891] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.777894] dracut-initqueue[583]: RTNETLINK answers: File exists [ 9.934728] random: crng init done [ 9.936079] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.353106] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.545746] printk: systemd: 24 output lines suppressed due to ratelimiting [ 11.830559] SELinux: Disabled at runtime. [ 11.889491] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.895468] systemd[1]: Detected virtualization kvm. [ 11.896752] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.401782] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.405276] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.411961] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.415585] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.420194] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.427266] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.430965] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ 12.543126] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.860743] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.178552] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.192663] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.340234] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.359368] EDAC sbridge: Ver: 1.1.2 [ 14.586988] Key type dns_resolver registered [ 14.890578] NFS: Registering the id_resolver key type [ 14.892717] Key type id_resolver registered [ 14.894652] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg246-server login: [ 60.572021] libcfs: loading out-of-tree module taints kernel. [ 60.614925] Key type ._llcrypt registered [ 60.618478] Key type .llcrypt registered [ 60.740244] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 80.323375] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 82.211221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 82.242846] alg: No test for adler32 (adler32-zlib) [ 84.182988] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 85.334260] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 87.169760] Key type lgssc registered [ 89.228742] Lustre: Echo OBD driver; http://www.lustre.org/ [ 105.471067] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 147.674095] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 163.316864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 163.357761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 164.824689] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 164.915367] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 165.110636] Lustre: lustre-MDT0000: new disk, initializing [ 165.251265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 165.290323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 170.007448] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 185.666808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 185.787734] Lustre: 6504:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 185.814866] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 185.819882] Lustre: Skipped 1 previous similar message [ 185.892847] Lustre: lustre-MDT0001: new disk, initializing [ 185.963723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 186.011972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 186.028373] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 192.204578] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 193.428081] hrtimer: interrupt took 3133114 ns [ 199.542383] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 212.073851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 212.479785] Lustre: lustre-OST0000: new disk, initializing [ 212.484342] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 212.488761] Lustre: 8409:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 212.546723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 219.297258] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.204979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 221.214418] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 221.330781] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 236.560811] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 236.753592] Lustre: lustre-OST0001: new disk, initializing [ 236.763120] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 236.769670] Lustre: 9465:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 236.829395] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 244.792745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 244.803033] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 245.403609] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 250.370366] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 260.500909] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 265.559478] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 271.896952] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing check_logdir /tmp/testlogs/ [ 277.619660] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing yml_node [ 282.481516] Lustre: DEBUG MARKER: Client: 2.17.53.3 [ 285.136829] Lustre: DEBUG MARKER: MDS: 2.17.53.3 [ 287.995897] Lustre: DEBUG MARKER: OSS: 2.17.53.3 [ 289.680913] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat May 16 05:17:30 EDT 2026 [ 310.910871] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 312.864060] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 331.752917] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 331.756815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 331.779800] Lustre: Skipped 3 previous similar messages [ 331.797579] Lustre: Skipped 3 previous similar messages [ 335.981097] Lustre: server umount lustre-MDT0000 complete [ 340.404471] LustreError: 6495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778923102 with bad export cookie 16778911442492299445 [ 340.408400] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 340.423964] LustreError: 6495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 341.985266] LustreError: 6510:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 341.986465] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 342.004913] LustreError: 6510:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 6 previous similar messages [ 342.015675] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 346.016718] LustreError: 6511:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 346.021774] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 346.028671] LustreError: 6511:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 346.037540] Lustre: Skipped 2 previous similar messages [ 346.851794] Lustre: server umount lustre-MDT0001 complete [ 351.358137] Lustre: server umount lustre-OST0000 complete [ 355.782927] Lustre: server umount lustre-OST0001 complete [ 364.004465] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 376.737099] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 421.016559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 421.661686] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 421.734086] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 421.914941] Lustre: lustre-MDT0000: new disk, initializing [ 422.149724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 422.185429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 427.998344] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 440.472900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 440.700437] Lustre: 14984:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 440.777966] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 440.784160] Lustre: Skipped 1 previous similar message [ 440.985090] Lustre: lustre-MDT0001: new disk, initializing [ 441.174500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 441.225302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 441.272733] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 446.560805] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 454.432138] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 458.702576] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 466.881909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 467.281924] Lustre: lustre-OST0000: new disk, initializing [ 467.289766] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 467.298476] Lustre: 16726:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 467.433837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 469.260844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 469.285331] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 469.477614] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 474.575709] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 483.106753] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 488.946756] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 489.213647] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 494.820333] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 495.060796] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 500.198127] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 500.199554] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 500.210071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 500.221070] Lustre: Skipped 2 previous similar messages [ 503.271916] Lustre: server umount lustre-OST0000 complete [ 507.873996] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 507.880553] LustreError: Skipped 1 previous similar message [ 507.883861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 507.894454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 507.897889] Lustre: Skipped 1 previous similar message [ 513.608882] Lustre: server umount lustre-MDT0000 complete [ 515.552948] LustreError: 14997:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 515.582163] LustreError: 14997:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 518.554853] LustreError: 14978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778923280 with bad export cookie 16778911442492300978 [ 518.558994] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 518.563286] LustreError: 14978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 518.835745] Lustre: server umount lustre-MDT0001 complete [ 526.205818] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 05:21:26 (1778923286) [ 548.027121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 548.804287] LustreError: 19457:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 548.950600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 553.957491] LustreError: 19458:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 554.045363] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 563.173226] LustreError: 19457:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 563.196031] LustreError: 19457:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 564.106865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 569.694641] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 575.282917] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 578.819161] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 580.944307] Lustre: Failing over lustre-MDT0000 [ 581.290936] Lustre: server umount lustre-MDT0000 complete [ 584.173511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 584.184407] Lustre: Skipped 2 previous similar messages [ 584.185745] LustreError: 20538:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 584.202918] LustreError: 20538:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 600.544229] Lustre: 3654:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778923346/real 1778923346] req@ffff8c333fa98000 x1865335803382144/t0(0) o400->MGC192.168.202.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1778923362 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 600.598942] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 602.749476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 611.186146] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 616.246312] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 616.422199] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 616.448362] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 616.496446] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 623.765402] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 628.022623] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 628.024323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 628.040540] Lustre: Skipped 1 previous similar message [ 628.061972] Lustre: Skipped 3 previous similar messages [ 634.408883] Lustre: server umount lustre-MDT0000 complete [ 635.363071] LustreError: 20567:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 635.376679] LustreError: 20567:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 13 previous similar messages [ 637.992461] LustreError: 19438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778923400 with bad export cookie 16778911442492302392 [ 638.014440] LustreError: 19438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 638.018052] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 638.418910] Lustre: server umount lustre-MDT0001 complete [ 647.612588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 648.355504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 648.366783] Lustre: Skipped 2 previous similar messages [ 652.882576] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 662.062167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 666.580361] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 672.168646] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 675.934299] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 678.112903] Lustre: Failing over lustre-MDT0000 [ 678.400693] Lustre: server umount lustre-MDT0000 complete [ 678.888225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 678.915823] Lustre: Skipped 1 previous similar message [ 698.009713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 698.237422] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 698.657919] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 703.728800] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 703.978679] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 703.991593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 704.017343] Lustre: Skipped 1 previous similar message [ 704.071821] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 713.228992] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 718.307037] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 718.310378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 718.320817] Lustre: Skipped 1 previous similar message [ 718.342232] Lustre: Skipped 3 previous similar messages [ 724.097108] Lustre: server umount lustre-MDT0000 complete [ 728.409117] LustreError: 22438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778923490 with bad export cookie 16778911442492303330 [ 728.418263] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 728.422355] LustreError: 22438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 728.847973] Lustre: server umount lustre-MDT0001 complete [ 738.407346] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 05:24:59 (1778923499) [ 757.694843] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 767.169405] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 810.829777] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 824.671798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 825.104620] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 825.171500] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 825.302925] Lustre: lustre-MDT0000: new disk, initializing [ 825.458544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 825.460472] Lustre: Skipped 2 previous similar messages [ 825.504161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 831.007518] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 843.948228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 844.092628] Lustre: 28444:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 844.476437] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 850.123395] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 859.637498] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 864.335429] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 873.448403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 873.950769] Lustre: 30187:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 875.291295] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 875.396467] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 875.524964] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 875.566758] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 881.611900] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 890.079673] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 897.165920] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 921.571538] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 921.573296] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 921.575726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 921.575733] Lustre: Skipped 2 previous similar messages [ 921.587320] Lustre: Skipped 1 previous similar message [ 927.802516] Lustre: server umount lustre-OST0000 complete [ 936.931961] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 939.659397] Lustre: server umount lustre-MDT0000 complete [ 942.056892] LustreError: 29302:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 942.077829] LustreError: 29302:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 14 previous similar messages [ 943.286050] LustreError: 29221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778923705 with bad export cookie 16778911442492303736 [ 943.291421] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 943.300582] LustreError: 29221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 943.728316] Lustre: server umount lustre-MDT0001 complete [ 955.066650] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 958.403215] Key type lgssc unregistered [ 958.670140] LNet: 32041:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.682082] LNetError: 32041:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 958.711172] LNet: Removed LNI 192.168.202.146@tcp [ 959.726167] Key type .llcrypt unregistered [ 959.728378] Key type ._llcrypt unregistered [ 983.334252] Key type ._llcrypt registered [ 983.335796] Key type .llcrypt registered [ 983.462775] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 985.192756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 985.212690] alg: No test for adler32 (adler32-zlib) [ 986.441538] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 986.777986] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 988.472601] Key type lgssc registered [ 989.932714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 999.518743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 999.544852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1001.057255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1005.470974] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1011.181262] LustreError: 33438:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1011.232027] LustreError: 33438:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1015.996756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1016.355537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1021.661777] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1028.570418] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1033.561763] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1041.765093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1042.471994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1046.625684] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1050.091453] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1057.456708] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1061.937199] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1077.818440] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 05:30:37 (1778923837) [ 1080.231336] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1082.426083] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1089.973154] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 05:30:50 (1778923850) [ 1101.777449] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 05:31:01 (1778923861) [ 1103.101650] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1113.182535] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1113.185935] Lustre: Skipped 2 previous similar messages [ 1114.864946] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 05:31:15 (1778923875) [ 1119.619809] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1119.624178] Lustre: Skipped 3 previous similar messages [ 1142.403755] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1142.412611] Lustre: Skipped 6 previous similar messages [ 1146.543632] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 05:31:46 (1778923906) [ 1150.555610] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1155.223874] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1166.773575] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 05:32:06 (1778923926) [ 1169.415466] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1169.425247] Lustre: Skipped 3 previous similar messages [ 1185.535927] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 05:32:25 (1778923945) [ 1205.220266] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1205.226794] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1205.249474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1205.270571] Lustre: Skipped 1 previous similar message [ 1208.528267] Lustre: server umount lustre-OST0000 complete [ 1215.468827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1215.480540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1215.495693] Lustre: Skipped 1 previous similar message [ 1215.515522] Lustre: Skipped 2 previous similar messages [ 1220.578463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1220.582749] Lustre: Skipped 1 previous similar message [ 1225.698727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1225.709138] Lustre: Skipped 1 previous similar message [ 1227.232121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1227.466974] Lustre: server umount lustre-MDT0000 complete [ 1230.818205] LustreError: 33438:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1230.848219] LustreError: 33438:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1232.791740] LustreError: 33419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778923994 with bad export cookie 5280433355265337148 [ 1232.802257] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1232.806247] LustreError: 33419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1233.087882] Lustre: server umount lustre-MDT0001 complete [ 1244.901935] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 1248.278815] Key type lgssc unregistered [ 1248.649788] LNet: 40943:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1248.668935] LNetError: 40943:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1248.695346] LNet: Removed LNI 192.168.202.146@tcp [ 1249.871330] Key type .llcrypt unregistered [ 1249.873444] Key type ._llcrypt unregistered [ 1270.611278] Key type ._llcrypt registered [ 1270.614310] Key type .llcrypt registered [ 1270.748934] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 1287.851700] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 1289.332699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1289.352159] alg: No test for adler32 (adler32-zlib) [ 1290.360238] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 1290.654863] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 1292.472175] Key type lgssc registered [ 1293.678038] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1336.714244] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1336.762219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1338.196045] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1338.275573] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1338.393248] Lustre: lustre-MDT0000: new disk, initializing [ 1338.599673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1338.631297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1343.740147] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1357.398424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1357.578359] Lustre: 44073:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1357.681850] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1357.690824] Lustre: Skipped 1 previous similar message [ 1357.767116] Lustre: lustre-MDT0001: new disk, initializing [ 1357.881571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1357.923467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1357.942703] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1363.346992] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1372.530978] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1377.448150] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1385.380212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1385.561216] Lustre: lustre-OST0000: new disk, initializing [ 1385.563644] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1385.567376] Lustre: 45814:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1385.642337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1393.029518] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1393.802252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1393.823112] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1393.979759] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1404.615960] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1409.234809] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1409.587360] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1413.266615] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1413.526593] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1419.748265] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1419.752869] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1419.770655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1419.804697] Lustre: Skipped 1 previous similar message [ 1421.315652] Lustre: server umount lustre-OST0000 complete [ 1429.481161] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1429.491733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1429.509488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1429.519297] Lustre: Skipped 1 previous similar message [ 1434.078719] Lustre: server umount lustre-MDT0000 complete [ 1438.501428] LustreError: 44066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778924200 with bad export cookie 16680625287215314225 [ 1438.509833] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1438.521310] LustreError: 44066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1439.103172] Lustre: server umount lustre-MDT0001 complete [ 1450.884677] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 05:36:51 (1778924211) [ 1459.291485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1460.080710] LustreError: 48128:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1460.130408] LustreError: 48128:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 1460.195569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1465.317126] LustreError: 48129:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1465.736570] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1470.438358] LustreError: 48128:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1475.366066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1475.554088] LustreError: 48129:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1475.869768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1480.758163] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1488.038477] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1493.671505] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1501.655795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1502.390506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1509.392988] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1516.117523] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1525.894300] LustreError: 50319:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1526.851288] Lustre: Modifying parameter general.timeout=1 in log params [ 1529.058444] Lustre: Modifying parameter general.timeout=3 in log params [ 1529.066465] Lustre: Skipped 1 previous similar message [ 1531.924762] Lustre: Modifying parameter general.timeout=5 in log params [ 1531.935780] Lustre: Skipped 1 previous similar message [ 1537.207871] Lustre: Modifying parameter general.timeout=8 in log params [ 1537.217813] Lustre: Skipped 2 previous similar messages [ 1545.436301] Lustre: Modifying parameter general.timeout=14 in log params [ 1545.446224] Lustre: Skipped 5 previous similar messages [ 1562.651606] Lustre: Modifying parameter general.timeout=29 in log params [ 1562.655697] Lustre: Skipped 14 previous similar messages [ 1595.272561] Lustre: Modifying parameter general.timeout=61 in log params [ 1595.281374] Lustre: Skipped 31 previous similar messages [ 1660.028573] Lustre: Modifying parameter general.timeout=121 in log params [ 1660.035102] Lustre: Skipped 59 previous similar messages [ 1680.645119] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 05:40:41 (1778924441) [ 1683.764237] Lustre: 57280:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1685.575209] Lustre: 57378:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1687.608492] Lustre: 57477:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1693.125195] Lustre: 57787:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1693.133776] Lustre: 57787:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 2 previous similar messages [ 1723.665596] Lustre: 59149:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1723.726701] LustreError: 59149:0:(mgs_handler.c:1202:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1728.139358] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 05:41:28 (1778924488) [ 1742.305792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1742.309247] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1742.336452] Lustre: Skipped 1 previous similar message [ 1742.342631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1742.353623] Lustre: Skipped 1 previous similar message [ 1748.422650] Lustre: server umount lustre-MDT0000 complete [ 1748.961670] LustreError: 49095:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1748.983475] LustreError: 49095:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1752.294751] LustreError: 48109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778924514 with bad export cookie 16680625287215315114 [ 1752.305560] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1752.314102] LustreError: 48109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1752.719554] Lustre: server umount lustre-MDT0001 complete [ 1767.512453] Lustre: server umount lustre-OST0000 complete [ 1799.512356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1799.797050] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1799.828684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1799.836823] Lustre: Skipped 31 previous similar messages [ 1800.067706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1804.086764] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1814.791645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1814.867640] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1814.883402] Lustre: 61682:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1820.486423] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1827.515072] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1832.811674] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1840.912740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1841.135050] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1841.386466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1841.389044] Lustre: Skipped 1 previous similar message [ 1847.923919] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1854.313911] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1856.042151] Lustre: 63909:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1867.238457] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1867.244441] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1867.251595] Lustre: Skipped 2 previous similar messages [ 1867.255955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1867.259959] Lustre: Skipped 3 previous similar messages [ 1869.696512] Lustre: server umount lustre-OST0000 complete [ 1876.961488] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1876.979392] LustreError: Skipped 1 previous similar message [ 1876.985318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1877.004103] Lustre: Skipped 1 previous similar message [ 1877.012295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1877.018683] Lustre: Skipped 1 previous similar message [ 1880.057576] Lustre: server umount lustre-MDT0000 complete [ 1883.112907] LustreError: 61689:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1883.135239] LustreError: 61689:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1884.426364] LustreError: 61673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778924646 with bad export cookie 16680625287215318362 [ 1884.431883] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1884.440651] LustreError: 61673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1884.777039] Lustre: server umount lustre-MDT0001 complete [ 1897.855825] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 1901.809805] Key type lgssc unregistered [ 1902.299441] LNet: 65097:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1902.322135] LNetError: 65097:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1902.346401] LNet: Removed LNI 192.168.202.146@tcp [ 1903.202879] Key type .llcrypt unregistered [ 1903.205325] Key type ._llcrypt unregistered [ 1930.641790] Key type ._llcrypt registered [ 1930.643707] Key type .llcrypt registered [ 1930.731996] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 1932.104986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1932.135514] alg: No test for adler32 (adler32-zlib) [ 1933.182693] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 1933.453686] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 1935.128180] Key type lgssc registered [ 1936.145727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1944.279513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1944.300203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1945.933426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1950.960298] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1955.321538] LustreError: 66494:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1955.352323] LustreError: 66494:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1960.439442] LustreError: 66495:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1961.277601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1962.098453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1967.378157] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1973.723309] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1977.766848] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1985.070545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1985.806565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1992.692312] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1999.094310] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2007.192155] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2015.737058] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2015.751695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2015.756717] Lustre: Skipped 1 previous similar message [ 2020.850488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2020.871910] Lustre: Skipped 4 previous similar messages [ 2021.404059] Lustre: server umount lustre-MDT0000 complete [ 2025.958470] LustreError: 66490:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2025.978804] LustreError: 66490:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2026.088921] LustreError: 67219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778924788 with bad export cookie 3055077449291681944 [ 2026.099451] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2026.101696] LustreError: 67219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2026.351727] Lustre: server umount lustre-MDT0001 complete [ 2041.083235] Lustre: server umount lustre-OST0000 complete [ 2070.377321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2070.619938] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2070.632356] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2070.747686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2074.673383] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2083.938279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2084.059645] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2084.085844] Lustre: 71031:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2084.570872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2088.653875] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2094.709838] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2098.003809] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2104.914822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2105.038100] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2105.369987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2111.773467] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2117.889904] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2119.599422] Lustre: 73264:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2121.510397] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2130.406069] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2130.416781] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2130.426030] Lustre: Skipped 1 previous similar message [ 2130.431332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2130.915173] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2132.391452] Lustre: server umount lustre-OST0000 complete [ 2136.036705] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2136.042250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2136.050195] Lustre: Skipped 2 previous similar messages [ 2136.057996] Lustre: Skipped 2 previous similar messages [ 2141.153570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2141.163768] Lustre: Skipped 1 previous similar message [ 2141.988409] Lustre: server umount lustre-MDT0000 complete [ 2145.398917] LustreError: 71759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778924907 with bad export cookie 3055077449291682763 [ 2145.406922] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2145.409536] LustreError: 71759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2145.757982] Lustre: server umount lustre-MDT0001 complete [ 2157.488869] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 2160.752190] Key type lgssc unregistered [ 2161.114164] LNet: 74457:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2161.121808] LNetError: 74457:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2162.155862] LNet: Removed LNI 192.168.202.146@tcp [ 2163.081503] Key type .llcrypt unregistered [ 2163.085179] Key type ._llcrypt unregistered [ 2187.322306] Key type ._llcrypt registered [ 2187.326192] Key type .llcrypt registered [ 2187.451789] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 2188.840359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2188.893086] alg: No test for adler32 (adler32-zlib) [ 2189.979347] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 2190.263894] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 2192.000161] Key type lgssc registered [ 2193.278760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2200.372356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2200.407996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2201.940929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2206.504427] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2211.309403] LustreError: 75853:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2211.328694] LustreError: 75853:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2214.435621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2214.882828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2219.619896] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2225.426988] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2229.195946] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2236.130920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2236.860793] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2242.977329] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2248.156662] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2254.549125] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2255.372419] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2256.395543] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2258.904581] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2258.908332] Lustre: Skipped 2 previous similar messages [ 2263.025561] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2263.032394] Lustre: Skipped 4 previous similar messages [ 2271.867727] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2271.872716] Lustre: Skipped 8 previous similar messages [ 2288.373757] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2288.379363] Lustre: Skipped 17 previous similar messages [ 2320.419780] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2320.427395] Lustre: Skipped 31 previous similar messages [ 2359.012921] Lustre: 83624:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2362.158960] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 05:52:02 (1778925122) [ 2363.781814] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2365.612226] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 05:52:06 (1778925126) [ 2373.091856] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2373.099928] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2373.108866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2374.113911] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2374.118965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2374.120843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2374.150833] Lustre: Skipped 1 previous similar message [ 2379.233428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2379.243118] Lustre: Skipped 1 previous similar message [ 2384.355441] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2384.372361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2384.386323] Lustre: Skipped 1 previous similar message [ 2384.561272] Lustre: server umount lustre-OST0000 complete [ 2388.961338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2388.967745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2388.983758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2394.802955] Lustre: server umount lustre-MDT0000 complete [ 2399.162737] LustreError: 75835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778925161 with bad export cookie 413618020226003068 [ 2399.171822] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2399.178898] LustreError: 75835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2399.518514] Lustre: server umount lustre-MDT0001 complete [ 2412.896499] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 2416.207930] Key type lgssc unregistered [ 2416.584117] LNet: 84857:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2416.587932] LNetError: 84857:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2416.608208] LNet: Removed LNI 192.168.202.146@tcp [ 2417.599649] Key type .llcrypt unregistered [ 2417.601691] Key type ._llcrypt unregistered [ 2422.372278] Key type ._llcrypt registered [ 2422.374825] Key type .llcrypt registered [ 2422.472481] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2426.196869] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules [ 2426.824479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2426.923657] alg: No test for adler32 (adler32-zlib) [ 2428.059561] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 2428.261235] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 2429.912150] Key type lgssc registered [ 2430.810955] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2432.294550] LustreError: 86025:0:(lu_object.c:1479:lu_context_key_register()) cfs_race id 60d sleeping [ 2449.838732] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 2455.203446] LustreError: 86025:0:(lu_object.c:1479:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2458.631462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2458.649451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2460.086725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2464.422379] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2470.376217] LustreError: 87032:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2470.403575] LustreError: 87032:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2472.900670] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 05:53:53 (1778925233) [ 2475.500304] LustreError: 87033:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2477.966422] LustreError: 87064:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2477.978211] LustreError: 87064:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2477.985208] LustreError: 87064:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 2478.455507] Lustre: server umount lustre-MDT0000 complete [ 2492.046901] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 2494.891551] Key type lgssc unregistered [ 2495.132980] LNet: 88522:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2495.140059] LNetError: 88522:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2495.157566] LNet: Removed LNI 192.168.202.146@tcp [ 2495.812063] Key type .llcrypt unregistered [ 2495.814623] Key type ._llcrypt unregistered [ 2511.800242] Key type ._llcrypt registered [ 2511.801297] Key type .llcrypt registered [ 2511.917983] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 2513.235185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2513.270519] alg: No test for adler32 (adler32-zlib) [ 2514.332049] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 2514.579572] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 2516.224178] Key type lgssc registered [ 2517.276592] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2524.767135] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2524.783236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2526.232540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2530.430174] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2536.420827] LustreError: 89736:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2536.438118] LustreError: 89736:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2539.293750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2539.961576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2544.429416] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2549.984960] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2552.862851] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2559.899953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2560.841293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2568.556107] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2574.702911] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2623.642356] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 05:56:24 (1778925384) [ 2625.336164] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2627.355231] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 05:56:27 (1778925387) [ 2632.679801] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2632.685622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2632.703878] Lustre: Skipped 2 previous similar messages [ 2632.707848] Lustre: Skipped 1 previous similar message [ 2637.797356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2637.807685] Lustre: Skipped 3 previous similar messages [ 2638.468342] Lustre: server umount lustre-MDT0000 complete [ 2642.568452] LustreError: 89717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778925404 with bad export cookie 6216369701451609297 [ 2642.575455] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2642.583805] LustreError: 89717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2642.932216] Lustre: server umount lustre-MDT0001 complete [ 2658.067264] Lustre: server umount lustre-OST0000 complete [ 2671.352149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2671.836819] LustreError: 93694:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2672.109058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2677.007038] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2677.221389] LustreError: 93695:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2677.240573] LustreError: 93695:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2681.328072] LustreError: 93694:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2686.366523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2686.433922] LustreError: 93695:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2686.995238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2692.324745] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2699.528168] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2704.724170] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2720.516565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2720.896876] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2720.910707] LustreError: 93684:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2720.946218] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2720.965366] LustreError: 95552:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2720.985844] LustreError: 95552:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2721.002578] LustreError: 95552:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2721.202323] Lustre: server umount lustre-OST0000 complete [ 2721.215330] LustreError: 95552:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2733.385630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2733.719091] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2733.737231] LustreError: 93683:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2733.767683] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2733.780096] LustreError: 96016:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2733.798085] LustreError: 96016:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2733.800171] LustreError: 96016:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2733.878492] Lustre: server umount lustre-OST0000 complete [ 2733.882860] LustreError: 96016:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2744.447231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2744.648426] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2744.657862] Lustre: Found index 0 for lustre-OST0000, updating log [ 2744.667361] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2744.787442] Lustre: lustre-OST0000: new disk, initializing [ 2744.790415] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2744.797262] Lustre: 96551:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2745.126407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2752.577101] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2756.913147] Lustre: Failing over lustre-OST0000 [ 2757.109311] Lustre: server umount lustre-OST0000 complete [ 2761.699813] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2761.700727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2761.733638] Lustre: Skipped 1 previous similar message [ 2761.756752] Lustre: Skipped 1 previous similar message [ 2766.820310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2766.832929] Lustre: Skipped 1 previous similar message [ 2767.815988] Lustre: server umount lustre-MDT0000 complete [ 2771.938256] LustreError: 93694:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2771.957760] LustreError: 93694:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2772.064537] LustreError: 93676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778925534 with bad export cookie 6216369701451612034 [ 2772.071531] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2772.083727] LustreError: 93676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2782.972715] Lustre: server umount lustre-MDT0001 complete [ 2792.306654] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 05:59:12 (1778925552) [ 2811.420310] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 2822.795714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2823.172956] LustreError: 98867:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2823.295287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2827.501494] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2837.452560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2843.288711] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2846.508799] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2854.367448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2855.501356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2855.504299] Lustre: Skipped 1 previous similar message [ 2855.524573] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2862.623534] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2862.696349] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2862.765677] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2862.779457] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2873.115065] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2873.196864] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2873.279238] Lustre: lustre-OST0001: new disk, initializing [ 2873.282679] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2873.287693] Lustre: 101135:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2880.659228] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2883.254783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2883.265308] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2883.396439] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2888.742684] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2900.669964] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2909.154697] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2909.163938] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2909.183322] Lustre: Skipped 1 previous similar message [ 2911.587741] Lustre: server umount lustre-MDT0001 complete [ 2914.276621] LustreError: 98862:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2914.289670] LustreError: 98862:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 2921.784880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2921.997934] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2922.001836] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2922.048168] Lustre: 98856:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2922.922413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2922.931597] Lustre: Skipped 1 previous similar message [ 2923.708396] LustreError: 102689:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2923.713155] LustreError: 102689:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2923.736843] LustreError: 98859:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2923.877986] LustreError: 102709:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.146@tcp: cfg command failed: rc = -17 [ 2923.894908] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2923.894908] [ 2927.184531] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2928.115103] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2939.103355] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 06:01:39 (1778925699) [ 2955.945205] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 2976.255193] Lustre: 104717:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2999.892916] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3003.484703] Lustre: 105851:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3096.553754] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3096.565070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3096.582677] Lustre: Skipped 1 previous similar message [ 3096.595947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3096.600018] Lustre: Skipped 3 previous similar messages [ 3102.186969] LustreError: 98860:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3102.199862] LustreError: 98860:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 8 previous similar messages [ 3103.201943] Lustre: server umount lustre-MDT0000 complete [ 3107.867367] LustreError: 98848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778925869 with bad export cookie 6216369701451613035 [ 3107.869925] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3107.882711] LustreError: 98848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3108.435606] Lustre: server umount lustre-MDT0001 complete [ 3124.015830] Lustre: server umount lustre-OST0000 complete [ 3138.994813] Lustre: server umount lustre-OST0001 complete [ 3146.076650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3158.223220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3168.551568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3180.296588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3201.321919] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 3213.601866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3213.624802] Lustre: lustre-MDT0000: reset Object Index mappings [ 3214.484902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3219.731739] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3229.584949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3229.624977] Lustre: lustre-MDT0001: reset Object Index mappings [ 3234.249229] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3237.749632] Lustre: 110903:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3246.640362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3255.191426] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3256.178108] LustreError: 111640:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3256.209570] LustreError: 111640:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 8 previous similar messages [ 3256.215220] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 3259.273431] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3265.932785] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3272.198677] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3272.201111] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3274.256509] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3282.532850] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3286.622190] Lustre: 112750:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3339.230575] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 06:08:19 (1778926099) [ 3368.417098] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3368.426134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3368.441127] Lustre: Skipped 3 previous similar messages [ 3368.447909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3368.453234] Lustre: Skipped 4 previous similar messages [ 3372.229410] Lustre: server umount lustre-MDT0000 complete [ 3376.330598] LustreError: 109770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778926138 with bad export cookie 6216369701451659872 [ 3376.342227] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3376.349118] LustreError: 109770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3379.689158] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3379.707719] Lustre: Skipped 4 previous similar messages [ 3382.747665] Lustre: server umount lustre-MDT0001 complete [ 3392.653346] Lustre: server umount lustre-OST0000 complete [ 3403.149306] Lustre: server umount lustre-OST0001 complete [ 3409.861266] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 3421.525744] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 3468.789141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3469.010643] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3469.031308] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3469.121608] Lustre: lustre-MDT0000: new disk, initializing [ 3469.286114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3475.498733] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3483.886935] Lustre: server umount lustre-MDT0000 complete [ 3494.722831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3495.079690] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3495.101334] Lustre: 116963:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3495.130235] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3495.335572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3495.340911] Lustre: Skipped 4 previous similar messages [ 3500.761687] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3507.076967] Lustre: server umount lustre-MDT0000 complete [ 3518.442767] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 06:11:18 (1778926278) [ 3520.482809] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3522.561180] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 06:11:23 (1778926283) [ 3542.253980] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 3554.471407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3560.954433] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3572.014672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3572.184850] Lustre: 118982:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3572.279859] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3572.283528] Lustre: Skipped 1 previous similar message [ 3572.377637] Lustre: lustre-MDT0001: new disk, initializing [ 3572.583344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3572.594961] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3578.366752] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3584.149375] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3591.543071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3591.871445] Lustre: lustre-OST0000: new disk, initializing [ 3591.878617] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3591.884668] Lustre: 120525:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3593.753497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3593.779061] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3593.999873] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3599.724328] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3611.070474] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3611.293648] Lustre: lustre-OST0001: new disk, initializing [ 3611.300922] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3611.311894] Lustre: 121379:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3612.918302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3612.934521] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3613.056261] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3619.352917] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3632.699980] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3639.131584] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4097.920726] Lustre: 127795:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4097.930142] Lustre: 127795:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4097.935246] Lustre: 127795:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4097.940049] Lustre: 127795:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4097.950676] Lustre: 127795:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4097.962471] Lustre: 127795:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4367.535604] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 06:25:28 (1778927128) [ 4383.718931] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4383.724456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4383.743077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4383.748495] Lustre: Skipped 8 previous similar messages [ 4387.383764] Lustre: server umount lustre-MDT0000 complete [ 4389.351210] LustreError: 122169:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4389.369070] LustreError: 122169:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 15 previous similar messages [ 4390.323131] LustreError: 122172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778927152 with bad export cookie 6216369701451727492 [ 4390.354529] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4394.468888] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4394.471766] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4394.475023] Lustre: Skipped 4 previous similar messages [ 4394.490699] Lustre: Skipped 4 previous similar messages [ 4396.840620] Lustre: server umount lustre-MDT0001 complete [ 4406.031252] Lustre: server umount lustre-OST0000 complete [ 4414.869091] Lustre: server umount lustre-OST0001 complete [ 4419.712576] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 4426.569840] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 4454.703933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4454.923150] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4454.939359] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4454.981379] Lustre: lustre-MDT0000: new disk, initializing [ 4455.065667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4455.071372] Lustre: Skipped 4 previous similar messages [ 4455.081738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4458.808502] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4467.724422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4467.802488] Lustre: 130806:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4467.845397] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4467.847925] Lustre: Skipped 1 previous similar message [ 4467.929742] Lustre: lustre-MDT0001: new disk, initializing [ 4468.024797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4468.032905] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4471.619667] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4478.217170] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4481.000877] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4486.704364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4486.952820] Lustre: lustre-OST0000: new disk, initializing [ 4486.955758] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4486.960309] Lustre: 132547:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4488.150185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4488.162033] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4488.293107] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4492.496166] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4499.274113] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4507.737186] Lustre: lustre-MDD0000: changelog on [ 4509.806863] Lustre: lustre-MDD0001: changelog on [ 5767.989511] Lustre: lustre-MDD0001: changelog off [ 5768.758525] Lustre: 136107:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5769.147053] Lustre: lustre-MDD0000: changelog off [ 5773.284860] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5773.290236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5773.290431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5773.296652] Lustre: Skipped 4 previous similar messages [ 5777.186422] Lustre: server umount lustre-OST0000 complete [ 5778.913900] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5778.916923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5778.921257] Lustre: Skipped 1 previous similar message [ 5778.924337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5783.522206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5783.522801] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5784.831116] Lustre: server umount lustre-MDT0000 complete [ 5786.240407] LustreError: 130800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778928548 with bad export cookie 6216369701452588758 [ 5786.244120] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5786.244956] LustreError: 130800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 5786.376615] Lustre: server umount lustre-MDT0001 complete [ 5790.102733] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 5791.264789] Key type lgssc unregistered [ 5791.412746] LNet: 141472:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5791.415753] LNetError: 141472:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5791.423485] LNet: Removed LNI 192.168.202.146@tcp [ 5791.743152] Key type .llcrypt unregistered [ 5791.744543] Key type ._llcrypt unregistered [ 5798.730849] Key type ._llcrypt registered [ 5798.732609] Key type .llcrypt registered [ 5798.773693] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 5805.216881] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 5805.573939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5805.642192] alg: No test for adler32 (adler32-zlib) [ 5806.516538] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 5806.604441] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 5808.192193] Key type lgssc registered [ 5808.654206] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5823.088750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5823.097479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5824.191953] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5824.207621] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5824.249118] Lustre: lustre-MDT0000: new disk, initializing [ 5824.300864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5824.309209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5825.895178] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5830.666643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5830.716780] Lustre: 144611:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5830.741406] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5830.744312] Lustre: Skipped 1 previous similar message [ 5830.782981] Lustre: lustre-MDT0001: new disk, initializing [ 5830.822990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5830.839695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5830.843701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5832.227781] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5835.651478] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5837.218578] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5840.237187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5840.356383] Lustre: lustre-OST0000: new disk, initializing [ 5840.359037] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5840.362559] Lustre: 146353:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5840.406910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5843.120885] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5846.542791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5846.548364] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5846.610098] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5846.764661] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5850.166654] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_module obdecho/obdecho [ 5850.887563] LustreError: 147228:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5850.890663] LustreError: 147228:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 5851.627380] LustreError: 147324:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 5856.738304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5856.740829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5856.742091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5856.742601] Lustre: Skipped 2 previous similar messages [ 5856.751147] Lustre: Skipped 2 previous similar messages [ 5861.420770] Lustre: server umount lustre-MDT0000 complete [ 5862.986320] LustreError: 144603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778928625 with bad export cookie 18436620444518004612 [ 5862.989811] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5862.991213] LustreError: 144603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5863.107581] Lustre: server umount lustre-MDT0001 complete [ 5874.975261] Lustre: server umount lustre-OST0000 complete [ 5882.964799] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 5886.762655] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 5900.458649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5900.551516] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5900.564978] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5900.602950] Lustre: lustre-MDT0000: new disk, initializing [ 5900.628845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5900.637329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5902.081735] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5906.503150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5906.547720] Lustre: 151000:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5906.571278] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5906.574216] Lustre: Skipped 1 previous similar message [ 5906.609661] Lustre: lustre-MDT0001: new disk, initializing [ 5906.645953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5906.663472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5906.670247] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5908.285176] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5911.773950] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5913.167122] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5915.757482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5915.879715] Lustre: lustre-OST0000: new disk, initializing [ 5915.881834] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5915.885059] Lustre: 152738:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5915.916593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5918.063115] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5918.592430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5918.596950] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5918.640638] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5921.487536] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5923.087326] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5923.167654] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5924.704040] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5924.785354] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5928.932702] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5928.933639] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5928.936216] LustreError: Skipped 1 previous similar message [ 5928.942985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5928.946298] Lustre: Skipped 1 previous similar message [ 5931.614263] Lustre: server umount lustre-OST0000 complete [ 5934.049552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5934.050680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5934.053741] Lustre: Skipped 2 previous similar messages [ 5939.170219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5939.173255] Lustre: Skipped 2 previous similar messages [ 5939.295585] Lustre: server umount lustre-MDT0000 complete [ 5940.798169] LustreError: 150991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778928702 with bad export cookie 18436620444518005571 [ 5940.803362] LustreError: 150991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5940.806657] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5940.929078] Lustre: server umount lustre-MDT0001 complete [ 5942.416471] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 06:51:43 (1778928703) [ 5943.031992] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 5943.761626] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 06:51:45 (1778928705) [ 5946.724940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5946.893507] LustreError: 154999:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5946.901833] LustreError: 154999:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 5946.919146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5948.356432] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5951.306689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5952.746907] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5954.576209] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5955.991789] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5958.390391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5960.569247] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5962.540267] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5977.059979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5977.060056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5977.062228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5977.070879] Lustre: Skipped 1 previous similar message [ 5981.729562] Lustre: server umount lustre-MDT0000 complete [ 5982.178068] LustreError: 154999:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5982.184394] LustreError: 154999:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5983.006039] LustreError: 154981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778928745 with bad export cookie 18436620444518006439 [ 5983.009352] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5983.010358] LustreError: 154981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5983.442882] LustreError: 154994:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5987.297798] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5987.298107] LustreError: 154994:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5987.304091] Lustre: Skipped 1 previous similar message [ 5987.312118] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5987.314981] Lustre: Skipped 4 previous similar messages [ 5989.176799] Lustre: server umount lustre-MDT0001 complete [ 5991.533942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5996.046837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5996.128169] LustreError: 158519:0:(llog.c:1646:llog_backup()) MGC192.168.202.146@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5996.131053] Lustre: 158519:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.202.146@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6013.137771] LustreError: 158524:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6013.143919] LustreError: 158524:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6023.137357] LustreError: 158524:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6023.144060] LustreError: 158524:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 6 previous similar messages [ 6039.762111] LustreError: 158526:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6039.768079] LustreError: 158526:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 14 previous similar messages [ 6057.056370] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6057.059277] Lustre: 158519:0:(mgc_request.c:1897:mgc_process_log()) MGC192.168.202.146@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6073.824855] LustreError: 158526:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6073.831387] LustreError: 158526:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 21 previous similar messages [ 6097.896342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6097.899284] Lustre: Skipped 2 previous similar messages [ 6099.290672] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6102.402953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6103.012216] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6104.036143] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xffdc089ce63357cf [ 6104.039491] Lustre: MGC192.168.202.146@tcp: Connection restored to 0@lo (at 0@lo) [ 6105.821919] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6107.948223] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6108.573861] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6109.161347] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6109.937372] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6110.655847] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6119.393428] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6119.394070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6119.396127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6119.396132] Lustre: Skipped 2 previous similar messages [ 6119.409428] Lustre: Skipped 2 previous similar messages [ 6122.926995] Lustre: server umount lustre-MDT0000 complete [ 6124.249274] LustreError: 154981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778928886 with bad export cookie 18436620444518012879 [ 6124.252399] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6124.254608] LustreError: 154981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6124.375957] Lustre: server umount lustre-MDT0001 complete [ 6135.898029] Lustre: server umount lustre-OST0000 complete [ 6144.267545] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 6147.795917] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 6161.471841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6161.558055] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6161.570634] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6161.607396] Lustre: lustre-MDT0000: new disk, initializing [ 6161.637591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6163.066399] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6167.865825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6167.908860] Lustre: 163331:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6168.002240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6168.005306] Lustre: Skipped 2 previous similar messages [ 6168.017726] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6169.719550] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6173.104825] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6174.451344] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6176.968484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6177.067218] Lustre: 165072:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6177.287115] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6177.322617] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6179.086549] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6182.207568] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6183.676318] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6183.752403] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6185.157580] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6185.237209] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6187.490991] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6187.491089] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6187.491863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6187.491868] Lustre: Skipped 2 previous similar messages [ 6187.496381] LustreError: Skipped 1 previous similar message [ 6187.499311] Lustre: Skipped 1 previous similar message [ 6200.288085] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6200.340363] Lustre: server umount lustre-OST0000 complete [ 6216.160118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6216.226110] Lustre: server umount lustre-MDT0000 complete [ 6217.497238] LustreError: 163323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778928979 with bad export cookie 18436620444518029672 [ 6217.500395] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6217.501148] LustreError: 163323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6217.609788] Lustre: server umount lustre-MDT0001 complete [ 6220.958679] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 06:56:22 (1778928982) [ 6223.296567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6223.445790] LustreError: 167336:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6223.452784] LustreError: 167336:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 23 previous similar messages [ 6224.831194] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6227.898379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6229.305528] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6231.223390] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6232.534200] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6235.121913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6237.487908] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6239.550770] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6254.406859] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 06:56:56 (1778929016) [ 6259.168973] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6259.171577] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6259.175930] Lustre: Skipped 2 previous similar messages [ 6259.178060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6259.179742] Lustre: Skipped 11 previous similar messages [ 6261.594410] Lustre: server umount lustre-OST0000 complete [ 6269.153734] Lustre: server umount lustre-MDT0000 complete [ 6270.498390] LustreError: 167316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778929032 with bad export cookie 18436620444518030547 [ 6270.501041] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6270.503573] LustreError: 167316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6270.614444] Lustre: server umount lustre-MDT0001 complete [ 6274.426063] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 6275.446673] Key type lgssc unregistered [ 6275.574274] LNet: 170730:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6275.577106] LNetError: 170730:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6275.583263] LNet: Removed LNI 192.168.202.146@tcp [ 6275.899664] Key type .llcrypt unregistered [ 6275.901467] Key type ._llcrypt unregistered [ 6283.607939] Key type ._llcrypt registered [ 6283.609117] Key type .llcrypt registered [ 6283.655327] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 6283.980336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6283.994753] alg: No test for adler32 (adler32-zlib) [ 6284.855424] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 6284.951330] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 6286.536193] Key type lgssc registered [ 6286.946649] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6289.672266] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6289.678301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6305.120443] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6310.240187] LustreError: 171932:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.146@tcp: failed processing log, type 4: rc = -110 [ 6315.362096] LustreError: 171932:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6315.367873] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6315.370397] LustreError: MGC192.168.202.146@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6315.375366] LustreError: 171932:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6315.379574] LustreError: 171932:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -22 [ 6315.383856] LustreError: 171932:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 6315.410587] Lustre: server umount lustre-OST0000 complete [ 6315.416395] LustreError: 171932:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6319.281228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6320.521606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6322.138300] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6325.871450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6326.010420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6327.607474] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6329.900363] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6331.468341] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6334.330082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6334.436148] LustreError: 174019:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6334.439943] Lustre: 174019:0:(mgc_request_server.c:748:mgc_llog_local_copy()) MGC192.168.202.146@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6334.488684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6336.669278] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6338.734771] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6341.601767] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6341.602211] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6341.603758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6341.605659] Lustre: Skipped 2 previous similar messages [ 6345.685311] Lustre: server umount lustre-MDT0000 complete [ 6346.721047] LustreError: 172432:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6346.726551] LustreError: 172432:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 6346.916263] LustreError: 172411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778929108 with bad export cookie 2433956733778356276 [ 6346.919241] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6346.919707] LustreError: 172411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6347.033351] Lustre: server umount lustre-MDT0001 complete [ 6358.625904] Lustre: server umount lustre-OST0000 complete [ 6361.918622] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 6365.393519] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 6376.801081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6376.886271] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6376.897976] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6376.928432] Lustre: lustre-MDT0000: new disk, initializing [ 6376.950027] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6376.956334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6378.168430] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6382.218962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6382.257189] Lustre: 177386:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6382.280174] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6382.282779] Lustre: Skipped 1 previous similar message [ 6382.321227] Lustre: lustre-MDT0001: new disk, initializing [ 6382.351954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6382.365236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6382.369670] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6383.690800] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6386.938165] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6388.186456] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6390.469872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6390.573032] Lustre: lustre-OST0000: new disk, initializing [ 6390.575100] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6390.578562] Lustre: 179128:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6390.613527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6392.580169] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6393.909445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6393.913314] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6393.942264] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6395.662474] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6397.007916] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6397.089800] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6398.445913] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6398.515815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6404.067982] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6404.068299] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6404.070166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6404.070170] Lustre: Skipped 2 previous similar messages [ 6404.070668] LustreError: Skipped 1 previous similar message [ 6404.073929] Lustre: Skipped 1 previous similar message [ 6405.268634] Lustre: server umount lustre-OST0000 complete [ 6408.160788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6408.163350] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6408.167295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6408.169499] Lustre: Skipped 1 previous similar message [ 6412.637476] Lustre: server umount lustre-MDT0000 complete [ 6413.818944] LustreError: 177379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778929175 with bad export cookie 2433956733778356976 [ 6413.821900] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6413.822884] LustreError: 177379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6413.930851] Lustre: server umount lustre-MDT0001 complete [ 6416.651471] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 06:59:38 (1778929178) [ 6422.397316] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 6423.348740] Key type lgssc unregistered [ 6423.473350] LNet: 181900:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6423.475647] LNetError: 181900:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6423.486332] LNet: Removed LNI 192.168.202.146@tcp [ 6423.771255] Key type .llcrypt unregistered [ 6423.772301] Key type ._llcrypt unregistered [ 6430.750882] Key type ._llcrypt registered [ 6430.752400] Key type .llcrypt registered [ 6430.792499] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 6431.220682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6431.229259] alg: No test for adler32 (adler32-zlib) [ 6432.088688] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 6432.168913] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 6433.752220] Key type lgssc registered [ 6434.141114] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6436.803603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6436.815221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6437.966773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6439.259304] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6442.318103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6442.440412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6443.785741] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6445.672573] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6446.861935] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6449.283740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6449.350489] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6449.351671] LustreError: 184712:0:(llog.c:1687:llog_backup()) MGC192.168.202.146@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6449.353970] Lustre: 184712:0:(mgc_request_server.c:745:mgc_llog_local_copy()) MGC192.168.202.146@tcp: can't backup local config lustre-OST0000: rc = -28 [ 6449.385442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6451.270596] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6453.276328] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6457.826530] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6457.828365] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6457.828594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6457.830828] Lustre: Skipped 1 previous similar message [ 6460.176573] Lustre: server umount lustre-MDT0000 complete [ 6461.383144] LustreError: 183057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778929223 with bad export cookie 2882870851643549251 [ 6461.386133] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6461.387500] LustreError: 183057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6461.489434] Lustre: server umount lustre-MDT0001 complete [ 6472.984539] Lustre: server umount lustre-OST0000 complete [ 6475.856602] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 6479.238602] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 6490.989850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6491.091023] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6491.105216] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6491.142966] Lustre: lustre-MDT0000: new disk, initializing [ 6491.171139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6491.177447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6492.605642] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6496.937724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6496.981080] Lustre: 188084:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6496.997848] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6496.999449] Lustre: Skipped 1 previous similar message [ 6497.001176] LustreError: 188851:0:(llog.c:1687:llog_backup()) MGC192.168.202.146@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 6497.005013] LustreError: 188851:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 6497.008216] Lustre: 188851:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.202.146@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 6497.016037] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6497.017707] Lustre: Skipped 1 previous similar message [ 6497.049824] Lustre: lustre-MDT0001: new disk, initializing [ 6497.082318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6497.094821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6497.099207] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6498.434784] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6501.684341] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6502.865368] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6505.190483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6505.287433] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6505.289234] Lustre: Skipped 1 previous similar message [ 6505.290771] LustreError: 189813:0:(llog.c:1687:llog_backup()) MGC192.168.202.146@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6505.293584] LustreError: 189813:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 6505.295619] Lustre: 189813:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.202.146@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 6505.299142] Lustre: 189813:0:(mgc_request_server.c:768:mgc_llog_local_copy()) Skipped 1 previous similar message [ 6505.308781] Lustre: lustre-OST0000: new disk, initializing [ 6505.310757] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6505.313976] Lustre: 189837:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6505.343139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6507.220141] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6510.302975] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6511.746328] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6514.678046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6514.681385] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6514.710723] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6514.870273] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 6516.141649] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6516.207297] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6519.778830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6519.779844] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6519.781105] LustreError: Skipped 1 previous similar message [ 6519.781689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6519.785790] Lustre: Skipped 2 previous similar messages [ 6519.786994] Lustre: Skipped 2 previous similar messages [ 6522.961919] Lustre: server umount lustre-OST0000 complete [ 6524.897236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6524.897700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6524.900376] Lustre: Skipped 1 previous similar message [ 6524.902953] Lustre: Skipped 1 previous similar message [ 6530.017418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6530.019699] Lustre: Skipped 1 previous similar message [ 6530.332743] Lustre: server umount lustre-MDT0000 complete [ 6531.619203] LustreError: 188076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778929293 with bad export cookie 2882870851643549937 [ 6531.622456] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6531.623324] LustreError: 188076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6531.743459] Lustre: server umount lustre-MDT0001 complete [ 6534.773278] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 07:01:36 (1778929296) [ 6540.136848] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 6543.555907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6543.683159] LustreError: 192813:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6543.690267] LustreError: 192813:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6543.700693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6544.986692] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6548.033077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6549.615865] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6550.665050] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6552.903371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6554.912333] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6557.825278] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6557.886132] Lustre: lustre-OST0001: new disk, initializing [ 6557.887851] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6557.890314] Lustre: 195069:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6559.794196] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6565.182929] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6566.514955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6566.518575] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6566.547742] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6569.855577] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6570.524811] Lustre: DEBUG MARKER: ADD OST3 [ 6572.351384] loop: module loaded [ 6572.371086] loop0: detected capacity change from 0 to 204800000 [ 6572.491099] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6572.494428] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6572.498034] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6574.723317] loop0: detected capacity change from 0 to 204800000 [ 6576.654481] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6576.713681] Lustre: lustre-OST0003: new disk, initializing [ 6576.715470] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 6576.717849] Lustre: 196775:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6576.757747] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 6576.760501] Lustre: Skipped 3 previous similar messages [ 6577.468694] Lustre: DEBUG MARKER: STOP OST3 [ 6578.158896] Lustre: Failing over lustre-OST0003 [ 6578.199915] Lustre: server umount lustre-OST0003 complete [ 6578.250134] LustreError: 194264:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6578.808226] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6582.719971] Lustre: DEBUG MARKER: START OST3 again [ 6583.001020] LustreError: 194265:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6583.006314] LustreError: 194265:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 6585.121734] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6585.219782] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 6587.030579] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 1 client reconnects [ 6587.039656] Lustre: lustre-OST0003: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 6587.130866] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6590.447130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 6590.451852] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 6590.460938] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 6600.780125] Lustre: Failing over lustre-OST0003 [ 6600.822156] Lustre: server umount lustre-OST0003 complete [ 6605.796344] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6605.796695] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6605.798948] LustreError: Skipped 1 previous similar message [ 6605.799650] LustreError: 194265:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6605.802735] Lustre: Skipped 1 previous similar message [ 6605.804238] LustreError: 194265:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6610.916548] LustreError: 194263:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6610.921290] LustreError: 194263:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 6615.008989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6615.011166] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6615.015500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6619.486099] Lustre: server umount lustre-MDT0000 complete [ 6620.617915] LustreError: 192794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778929382 with bad export cookie 2882870851643550819 [ 6620.620568] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6620.620774] LustreError: 192794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6621.152831] LustreError: 194447:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6621.158020] LustreError: 194447:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6625.185208] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6625.187030] Lustre: Skipped 7 previous similar messages [ 6626.931419] Lustre: server umount lustre-MDT0001 complete [ 6628.110744] Lustre: server umount lustre-OST0000 complete [ 6629.270448] Lustre: server umount lustre-OST0001 complete [ 6631.204919] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 6633.694898] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 6645.669604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6645.756274] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6645.767050] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6645.800256] Lustre: lustre-MDT0000: new disk, initializing [ 6645.824268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6645.826283] Lustre: Skipped 1 previous similar message [ 6645.831940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6647.129389] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6651.439324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6651.481266] Lustre: 202114:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6651.582028] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6652.839920] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6655.908303] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6657.019812] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6659.211630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6659.304822] Lustre: 203858:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6660.669525] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6660.713454] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6661.278897] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6664.548511] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6666.117499] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6666.200637] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6667.812313] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6667.902716] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6670.819268] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6670.819527] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6670.822311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6670.829187] Lustre: Skipped 6 previous similar messages [ 6674.774210] Lustre: server umount lustre-OST0000 complete [ 6682.201733] Lustre: server umount lustre-MDT0000 complete [ 6683.428071] LustreError: 202106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778929445 with bad export cookie 2882870851643552989 [ 6683.431540] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6683.432486] LustreError: 202106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6683.536606] Lustre: server umount lustre-MDT0001 complete [ 6684.693221] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 07:04:06 (1778929446) [ 6689.574822] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 6692.496256] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 6704.322501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6704.413371] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6704.425952] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6704.427505] Lustre: Skipped 3 previous similar messages [ 6704.464769] Lustre: lustre-MDT0000: new disk, initializing [ 6704.466098] Lustre: Skipped 2 previous similar messages [ 6704.494785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6704.498511] Lustre: Skipped 2 previous similar messages [ 6705.784100] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6709.815768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6709.853330] Lustre: 208276:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6709.941971] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6711.216353] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6714.268019] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6715.410780] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6718.067166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6718.173424] Lustre: 210018:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6719.261056] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6720.072503] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6723.174083] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6724.549931] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6724.619554] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6726.042881] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6726.124284] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6729.700663] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6729.701248] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6729.706560] Lustre: Skipped 2 previous similar messages [ 6729.708765] LustreError: Skipped 1 previous similar message [ 6729.711420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6729.714901] Lustre: Skipped 5 previous similar messages [ 6732.880908] Lustre: server umount lustre-OST0000 complete [ 6740.252133] Lustre: server umount lustre-MDT0000 complete [ 6741.462081] LustreError: 208270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778929503 with bad export cookie 2882870851643553871 [ 6741.464563] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6741.465428] LustreError: 208270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6741.575552] Lustre: server umount lustre-MDT0001 complete [ 6744.267150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6744.408864] LustreError: 212085:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6744.414212] LustreError: 212085:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6745.608930] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6748.421275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6749.881098] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6751.681828] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6752.854341] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6755.020710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6756.951331] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6758.690700] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6778.848770] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6778.851052] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6778.855148] Lustre: Skipped 3 previous similar messages [ 6781.579886] Lustre: server umount lustre-OST0000 complete [ 6795.233907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6795.236797] Lustre: Skipped 12 previous similar messages [ 6797.280075] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6797.343750] Lustre: server umount lustre-MDT0000 complete [ 6798.583193] LustreError: 212067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778929560 with bad export cookie 2882870851643554753 [ 6798.586380] LustreError: 212067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6798.586439] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6798.698625] Lustre: server umount lustre-MDT0001 complete [ 6891.495123] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 6892.410585] Key type lgssc unregistered [ 6892.532539] LNet: 215412:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6892.535846] LNetError: 215412:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6892.548424] LNet: Removed LNI 192.168.202.146@tcp [ 6892.839105] Key type .llcrypt unregistered [ 6892.840659] Key type ._llcrypt unregistered [ 6900.224716] Key type ._llcrypt registered [ 6900.225653] Key type .llcrypt registered [ 6900.257090] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 6905.991572] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 6906.367808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6906.383353] alg: No test for adler32 (adler32-zlib) [ 6907.226757] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 6907.309986] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 6908.888105] Key type lgssc registered [ 6909.223812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6919.803726] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6919.811027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6920.900410] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6920.912431] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6920.950410] Lustre: lustre-MDT0000: new disk, initializing [ 6920.972659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6920.978288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6922.248642] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6926.215749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6926.253468] Lustre: 218705:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6926.274795] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6926.277232] Lustre: Skipped 1 previous similar message [ 6926.312578] Lustre: lustre-MDT0001: new disk, initializing [ 6926.343762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6926.354453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6926.357385] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6927.531218] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6930.454045] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6931.528811] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6933.605340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6933.705930] Lustre: lustre-OST0000: new disk, initializing [ 6933.707355] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6933.709771] Lustre: 220445:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6933.742763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6935.098919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6935.101761] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6935.138437] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6935.664956] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6938.651674] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6939.915342] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6939.983110] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6941.221566] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6941.293784] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6945.250915] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6945.251236] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6945.254878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6945.256277] LustreError: Skipped 1 previous similar message [ 6945.260454] Lustre: Skipped 1 previous similar message [ 6947.984670] Lustre: server umount lustre-OST0000 complete [ 6950.369570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6950.370225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6950.373235] Lustre: Skipped 2 previous similar messages [ 6955.357653] Lustre: server umount lustre-MDT0000 complete [ 6956.578480] LustreError: 219481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778929718 with bad export cookie 559340809540057883 [ 6956.581238] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6956.587246] LustreError: 219481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6956.699237] Lustre: server umount lustre-MDT0001 complete [ 6962.558150] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 6965.983301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6966.128733] LustreError: 223208:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6966.136481] LustreError: 223208:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 6966.143785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6967.430145] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6970.229921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6970.347029] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6971.611705] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6972.515591] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6974.556835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6976.364129] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6979.294928] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6979.362903] Lustre: lustre-OST0001: new disk, initializing [ 6979.364674] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6979.368900] Lustre: 225460:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6979.407911] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6979.410415] Lustre: Skipped 1 previous similar message [ 6980.736590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6980.740353] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6980.767943] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6981.351028] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6986.846834] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6991.478313] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6995.126558] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 07:09:16 (1778929756) [ 6995.937170] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6995.939935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6995.943780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6995.945362] Lustre: Skipped 1 previous similar message [ 7001.570121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7001.572253] Lustre: Skipped 7 previous similar messages [ 7001.943694] Lustre: server umount lustre-MDT0000 complete [ 7003.084210] LustreError: 226272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778929765 with bad export cookie 559340809540058751 [ 7003.087667] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7003.088912] LustreError: 226272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7003.197289] Lustre: server umount lustre-MDT0001 complete [ 7014.913725] Lustre: server umount lustre-OST0000 complete [ 7026.333694] Lustre: server umount lustre-OST0001 complete [ 7028.289547] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 7031.433419] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 7042.753137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7042.834284] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7042.846813] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7042.879960] Lustre: lustre-MDT0000: new disk, initializing [ 7042.902263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7042.908254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7044.166043] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7048.190675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7048.227810] Lustre: 229349:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7048.322404] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7049.609900] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7052.534589] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7053.641854] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7055.745173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7055.829315] Lustre: lustre-OST0000: new disk, initializing [ 7055.830565] Lustre: Skipped 1 previous similar message [ 7055.832081] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7055.833616] Lustre: Skipped 2 previous similar messages [ 7055.835387] Lustre: 231091:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7057.574019] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7057.719925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7057.722488] Lustre: Skipped 1 previous similar message [ 7057.724138] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7057.762740] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7060.566726] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7061.875063] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7061.944495] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7063.220655] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7063.296145] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7068.130660] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7068.131036] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7068.132951] LustreError: Skipped 1 previous similar message [ 7068.133595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7068.137318] Lustre: Skipped 4 previous similar messages [ 7068.141713] Lustre: Skipped 1 previous similar message [ 7078.368123] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7078.370441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7078.373477] Lustre: Skipped 3 previous similar messages [ 7078.418091] Lustre: server umount lustre-OST0000 complete [ 7083.490781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7083.495493] Lustre: Skipped 1 previous similar message [ 7085.654112] Lustre: server umount lustre-MDT0000 complete [ 7086.982484] LustreError: 229341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778929849 with bad export cookie 559340809540060844 [ 7086.986534] LustreError: 229341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7086.987502] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7087.114921] Lustre: server umount lustre-MDT0001 complete [ 7090.078964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7090.252826] LustreError: 233158:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7090.279094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7090.281134] Lustre: Skipped 2 previous similar messages [ 7091.602499] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7094.509726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7096.062806] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7097.944467] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7099.274519] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7101.670809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7103.835476] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7105.703717] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7253.477749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7253.480682] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7253.481049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7253.481061] Lustre: Skipped 2 previous similar messages [ 7253.481332] Lustre: Skipped 1 previous similar message [ 7253.483458] LustreError: Skipped 1 previous similar message [ 7255.122203] Lustre: server umount lustre-OST0000 complete [ 7258.594743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7262.492164] Lustre: server umount lustre-MDT0000 complete [ 7263.713594] LustreError: 233159:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7263.717623] LustreError: 233159:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 7263.787098] LustreError: 233139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778930025 with bad export cookie 559340809540061726 [ 7263.790520] LustreError: 233139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7263.791116] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7263.913891] Lustre: server umount lustre-MDT0001 complete [ 7370.157904] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 7371.332166] Key type lgssc unregistered [ 7371.475364] LNet: 236407:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7371.478044] LNetError: 236407:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7371.487348] LNet: Removed LNI 192.168.202.146@tcp [ 7371.846125] Key type .llcrypt unregistered [ 7371.847411] Key type ._llcrypt unregistered [ 7379.538539] Key type ._llcrypt registered [ 7379.539359] Key type .llcrypt registered [ 7379.570541] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 7385.392217] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 7385.877779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7385.885708] alg: No test for adler32 (adler32-zlib) [ 7386.750336] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 7386.841412] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 7388.424198] Key type lgssc registered [ 7388.808692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7403.700028] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 7407.124267] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7407.129368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7408.217858] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7408.229438] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7408.273031] Lustre: lustre-MDT0000: new disk, initializing [ 7408.299849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7408.305545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7409.609952] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7413.779415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7413.821539] Lustre: 240395:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7413.844478] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7413.846908] Lustre: Skipped 1 previous similar message [ 7413.886391] Lustre: lustre-MDT0001: new disk, initializing [ 7413.919908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7413.932827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7413.937497] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7415.235964] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7417.531636] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7419.782424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7419.883515] Lustre: lustre-OST0000: new disk, initializing [ 7419.885432] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7419.888097] Lustre: 241991:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7419.917403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7421.840246] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7425.021063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7425.023869] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7425.062909] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7426.105036] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7426.174549] Lustre: lustre-OST0001: new disk, initializing [ 7426.176724] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7426.179432] Lustre: 242849:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7426.212217] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7428.190793] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7433.843600] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7434.245120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7434.248886] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7434.285666] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7440.691737] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7444.450192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7444.452540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7444.453800] Lustre: Skipped 3 previous similar messages [ 7444.455930] Lustre: Skipped 2 previous similar messages [ 7449.569171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7449.571164] Lustre: Skipped 2 previous similar messages [ 7449.760176] Lustre: server umount lustre-MDT0000 complete [ 7450.923080] LustreError: 241172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778930212 with bad export cookie 1864544599022176462 [ 7450.926099] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7450.927408] LustreError: 241172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7451.032119] Lustre: server umount lustre-MDT0001 complete [ 7462.918731] Lustre: server umount lustre-OST0000 complete [ 7474.400606] Lustre: server umount lustre-OST0001 complete [ 7478.719577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7481.846528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7484.697203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7487.807083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7494.044831] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 7497.561654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7497.568457] Lustre: lustre-MDT0000: reset Object Index mappings [ 7497.760092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7499.160659] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7502.044524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7502.051053] Lustre: lustre-MDT0001: reset Object Index mappings [ 7503.498899] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7504.474502] Lustre: 248921:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7506.721209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7506.862361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7506.866052] Lustre: Skipped 1 previous similar message [ 7507.876077] LustreError: 249273:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7507.880097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7507.884633] LustreError: 249273:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 7508.804839] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7511.490986] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7512.617344] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 7513.325994] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7516.419235] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7517.646667] Lustre: 250752:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7522.786479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7522.789830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7522.791368] Lustre: Skipped 2 previous similar messages [ 7522.792954] Lustre: Skipped 2 previous similar messages [ 7522.795789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7525.027439] Lustre: server umount lustre-MDT0000 complete [ 7526.288284] LustreError: 248541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778930288 with bad export cookie 1864544599022183238 [ 7526.292494] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7526.293524] LustreError: 248541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7526.407757] Lustre: server umount lustre-MDT0001 complete [ 7537.097120] Lustre: server umount lustre-OST0000 complete [ 7548.578504] Lustre: server umount lustre-OST0001 complete [ 7555.008316] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 07:18:36 (1778930316) [ 7560.333830] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 7563.798128] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 7575.733420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7575.825069] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7575.835095] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7575.869573] Lustre: lustre-MDT0000: new disk, initializing [ 7575.892173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7575.894127] Lustre: Skipped 1 previous similar message [ 7575.899240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7577.174256] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7581.344710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7581.393932] Lustre: 254968:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7581.498218] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7582.824360] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7585.982054] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7587.202353] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7589.506889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7589.616474] Lustre: lustre-OST0000: new disk, initializing [ 7589.617921] Lustre: Skipped 1 previous similar message [ 7589.619432] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7589.621452] Lustre: Skipped 2 previous similar messages [ 7589.623274] Lustre: 256708:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7590.850803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7590.853315] Lustre: Skipped 1 previous similar message [ 7590.855108] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7590.897756] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7591.637737] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7594.777308] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7596.240946] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7596.319063] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7597.689392] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7597.767839] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7599.010398] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7599.012215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7599.014778] Lustre: Skipped 2 previous similar messages [ 7599.016641] Lustre: Skipped 2 previous similar messages [ 7604.194329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7604.197399] Lustre: Skipped 2 previous similar messages [ 7604.492838] Lustre: server umount lustre-OST0000 complete [ 7607.264903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7607.269275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7614.433328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7614.435743] Lustre: Skipped 4 previous similar messages [ 7620.064090] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7620.127329] Lustre: server umount lustre-MDT0000 complete [ 7621.403440] LustreError: 255746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778930383 with bad export cookie 1864544599022185317 [ 7621.406679] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7621.407690] LustreError: 255746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7621.530552] Lustre: server umount lustre-MDT0001 complete [ 7627.396164] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 7630.547529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7630.687961] LustreError: 259470:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7630.702648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7630.704619] Lustre: Skipped 2 previous similar messages [ 7632.017072] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7634.827820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7636.266609] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7637.202070] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7639.558078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7641.382115] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7644.018591] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7644.085691] Lustre: lustre-OST0001: new disk, initializing [ 7644.087845] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7644.090869] Lustre: 261724:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7645.692551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7645.696906] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7645.724192] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7645.910126] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7650.283904] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7651.381587] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7652.974406] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 7652.979174] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 7655.393326] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7655.396417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7655.399918] Lustre: Skipped 1 previous similar message [ 7655.402060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7655.404655] Lustre: Skipped 2 previous similar messages [ 7660.884991] Lustre: server umount lustre-MDT0000 complete [ 7661.026133] LustreError: 259467:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7661.031719] LustreError: 259467:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 7661.963921] LustreError: 260572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778930424 with bad export cookie 1864544599022186199 [ 7661.966560] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7661.966779] LustreError: 260572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7662.069219] Lustre: server umount lustre-MDT0001 complete [ 7673.348734] Lustre: server umount lustre-OST0000 complete [ 7684.849136] Lustre: server umount lustre-OST0001 complete [ 7690.524924] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 7693.770813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7693.896503] LustreError: 264740:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7695.152199] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7698.207280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7698.341684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7698.343827] Lustre: Skipped 4 previous similar messages [ 7699.748691] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7700.821550] Lustre: 265846:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7703.067713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7704.228448] LustreError: 266199:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7704.234578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 7705.134809] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7708.070473] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7709.179829] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 7710.124000] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7713.336485] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7714.641538] Lustre: 267688:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7716.431439] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 7716.437072] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 7719.576622] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 07:21:21 (1778930481) [ 7734.240912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7734.243161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7734.246824] Lustre: Skipped 3 previous similar messages [ 7734.248553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7734.250252] Lustre: Skipped 3 previous similar messages [ 7737.178368] Lustre: server umount lustre-MDT0000 complete [ 7738.415052] LustreError: 265466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778930500 with bad export cookie 1864544599022192499 [ 7738.418227] LustreError: 265466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7738.418461] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7739.872956] LustreError: 266382:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7739.879082] LustreError: 266382:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 7744.628314] Lustre: server umount lustre-MDT0001 complete [ 7760.352092] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7760.391367] Lustre: server umount lustre-OST0000 complete [ 7767.704266] Lustre: server umount lustre-OST0001 complete [ 7769.806110] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 7773.109548] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 7788.513663] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 7791.739916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7791.825974] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7791.837568] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7791.879504] Lustre: lustre-MDT0000: new disk, initializing [ 7791.910423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7793.159158] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7797.109735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7797.150597] Lustre: 271871:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7797.245319] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7798.463781] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7802.790348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7802.887350] Lustre: 273469:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7804.194397] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7804.778330] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7808.958701] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7811.050263] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7811.056505] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7815.529232] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7820.899553] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 07:23:02 (1778930582) [ 7822.817063] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7822.819233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7822.822371] Lustre: Skipped 5 previous similar messages [ 7822.823811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7822.825482] Lustre: Skipped 7 previous similar messages [ 7827.745400] Lustre: server umount lustre-MDT0000 complete [ 7828.928261] LustreError: 272648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778930590 with bad export cookie 1864544599022200367 [ 7828.931644] LustreError: 272648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7828.931865] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7829.039045] Lustre: server umount lustre-MDT0001 complete [ 7840.709618] Lustre: server umount lustre-OST0000 complete [ 7852.196621] Lustre: server umount lustre-OST0001 complete [ 7854.979279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7855.137682] LustreError: 276548:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7855.144127] LustreError: 276548:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 7855.172162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7855.174827] Lustre: Skipped 6 previous similar messages [ 7856.570910] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7859.468612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7860.971034] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7862.718842] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7864.039760] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7866.302947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7868.298994] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7870.121170] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7887.281281] loop1: detected capacity change from 0 to 204800000 [ 7887.360330] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7887.364478] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7887.368786] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7889.100877] loop1: detected capacity change from 0 to 204800000 [ 7890.915080] LustreError: 278150:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7890.922134] LustreError: 278150:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 10 previous similar messages [ 7890.966106] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7890.996136] LustreError: 276535:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs11695: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 7891.001277] LustreError: 276535:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs11695 [ 7891.003334] LustreError: 276535:0:(mgs_handler.c:517:mgs_target_reg()) fs11695-MDT0001: Error checking target index: rc = -13 [ 7891.006942] LustreError: fs11695-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 7891.010231] LustreError: 279997:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [ 7891.012740] LustreError: 279997:0:(tgt_mount.c:2084:server_put_super()) no obd fs11695-MDT0001 [ 7891.014978] LustreError: 279997:0:(tgt_mount.c:116:server_deregister_mount()) fs11695-MDT0001 not registered [ 7891.024815] Lustre: server umount fs11695-MDT0001 complete [ 7891.026352] LustreError: 279997:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 7893.824403] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7893.850985] Lustre: Setting parameter fs11695-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs11695-MDT0001 [ 7893.854192] Lustre: Skipped 2 previous similar messages [ 7893.870785] Lustre: srv-fs11695-MDT0001: No data found on store. Initialize space. [ 7893.874189] Lustre: Skipped 4 previous similar messages [ 7893.911655] Lustre: fs11695-MDT0001: new disk, initializing [ 7893.913013] Lustre: Skipped 3 previous similar messages [ 7893.951619] Lustre: srv-fs11695-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7893.953722] Lustre: 280461:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs11695-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7893.956074] LustreError: 280461:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs11695-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7895.318953] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7898.278850] Lustre: server umount fs11695-MDT0001 complete [ 7902.537866] loop1: detected capacity change from 0 to 204800000 [ 7904.416177] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7904.540069] Lustre: 282009:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs11695-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7904.543319] LustreError: 282009:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs11695-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7905.849158] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7907.502597] Lustre: server umount fs11695-MDT0001 complete [ 7910.880803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7910.882818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7910.885977] Lustre: Skipped 3 previous similar messages [ 7916.841253] Lustre: server umount lustre-MDT0000 complete [ 7919.746922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7919.794437] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7921.204822] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7923.686487] loop1: detected capacity change from 0 to 204800000 [ 7925.227943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7925.584045] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7925.702283] Lustre: 284416:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs11695-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7925.705898] LustreError: 284416:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs11695-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7927.028503] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7928.733483] Lustre: server umount fs11695-MDT0001 complete [ 7935.969127] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7941.654757] Lustre: server umount lustre-OST0000 complete [ 7949.085968] Lustre: server umount lustre-MDT0000 complete [ 7950.329386] LustreError: 276530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778930712 with bad export cookie 1864544599022203041 [ 7950.333126] LustreError: 276530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7950.455241] Lustre: server umount lustre-MDT0001 complete [ 7954.468484] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 7955.526201] Key type lgssc unregistered [ 7955.659358] LNet: 286721:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7955.662062] LNetError: 286721:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7955.668419] LNet: Removed LNI 192.168.202.146@tcp [ 7955.984099] Key type .llcrypt unregistered [ 7955.985169] Key type ._llcrypt unregistered [ 7964.059072] Key type ._llcrypt registered [ 7964.060071] Key type .llcrypt registered [ 7964.098135] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 7964.512331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7964.520383] alg: No test for adler32 (adler32-zlib) [ 7965.380967] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 7965.473734] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 7967.056148] Key type lgssc registered [ 7967.460721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7970.224680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7970.230796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7971.422726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7972.836493] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7975.709041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7975.853982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7977.177648] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7978.981770] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7980.212334] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7982.423203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7982.550759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7984.546878] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7985.638951] LustreError: 289553:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7985.644834] LustreError: 289553:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 7986.309288] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7990.757278] LustreError: 289553:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7995.874939] LustreError: 289554:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8001.752618] LustreError: 289553:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8001.760261] LustreError: 289553:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 8006.868258] LustreError: 289553:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8006.873236] LustreError: 289553:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 8008.083142] loop1: detected capacity change from 0 to 204800000 [ 8008.146986] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8008.151813] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8008.155897] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8009.860481] loop1: detected capacity change from 0 to 204800000 [ 8011.798755] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8011.858265] Lustre: fs19201-OST0000: new disk, initializing [ 8011.860263] Lustre: srv-fs19201-OST0000: No data found on store. Initialize space. [ 8011.862821] Lustre: 291414:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8011.891061] Lustre: fs19201-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8013.897776] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8017.107459] LustreError: 290049:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8017.116182] LustreError: 290049:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 8017.403972] Lustre: server umount fs19201-OST0000 complete [ 8021.458129] loop1: detected capacity change from 0 to 204800000 [ 8023.687840] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8023.778636] Lustre: fs19201-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8025.855024] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8028.714485] loop2: detected capacity change from 0 to 204800000 [ 8028.805151] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8028.809729] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8028.814718] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8030.629693] loop2: detected capacity change from 0 to 204800000 [ 8032.566267] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8032.598264] LustreError: fs19201-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 8032.604176] LustreError: fs19201-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 8032.607803] LustreError: 294417:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [ 8032.610205] LustreError: 294417:0:(tgt_mount.c:2084:server_put_super()) no obd fs19201-OST0001 [ 8032.612565] LustreError: 294417:0:(tgt_mount.c:116:server_deregister_mount()) fs19201-OST0001 not registered [ 8032.623896] Lustre: server umount fs19201-OST0001 complete [ 8032.625277] LustreError: 294417:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 8036.115197] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8036.158561] Lustre: fs19201-OST0001: new disk, initializing [ 8036.160137] Lustre: srv-fs19201-OST0001: No data found on store. Initialize space. [ 8036.162528] Lustre: 294967:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8036.184452] Lustre: fs19201-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8037.586815] LustreError: 289555:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8037.594831] LustreError: 289555:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 11 previous similar messages [ 8038.088136] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8041.591328] Lustre: server umount fs19201-OST0000 complete [ 8044.844664] Lustre: server umount fs19201-OST0001 complete [ 8049.027874] loop1: detected capacity change from 0 to 204800000 [ 8050.959103] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8052.988664] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8055.290409] loop2: detected capacity change from 0 to 204800000 [ 8057.202943] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8057.263685] Lustre: fs19201-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8057.265847] Lustre: Skipped 1 previous similar message [ 8059.198446] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8062.143089] Lustre: server umount fs19201-OST0000 complete [ 8065.296833] Lustre: server umount fs19201-OST0001 complete [ 8072.672957] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8072.675099] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8072.678819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8073.699089] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8073.699368] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8073.700192] LustreError: 289555:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8073.700212] LustreError: 289555:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 20 previous similar messages [ 8073.700714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8073.717515] LustreError: Skipped 1 previous similar message [ 8075.353195] Lustre: server umount lustre-OST0000 complete [ 8078.304896] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8078.307400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8078.311144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8078.312982] Lustre: Skipped 1 previous similar message [ 8082.781061] Lustre: server umount lustre-MDT0000 complete [ 8083.997254] LustreError: 299788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778930846 with bad export cookie 14338002687373824792 [ 8084.000553] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8084.001342] LustreError: 299788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8084.118795] Lustre: server umount lustre-MDT0001 complete [ 8087.952769] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 8088.954757] Key type lgssc unregistered [ 8089.077389] LNet: 300810:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8089.081151] LNetError: 300810:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8089.090315] LNet: Removed LNI 192.168.202.146@tcp [ 8089.398104] Key type .llcrypt unregistered [ 8089.399102] Key type ._llcrypt unregistered [ 8099.758322] Key type ._llcrypt registered [ 8099.759574] Key type .llcrypt registered [ 8099.801787] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 8105.999195] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 8106.331900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8106.382447] alg: No test for adler32 (adler32-zlib) [ 8107.239282] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 8107.326838] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 8108.912167] Key type lgssc registered [ 8109.307381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8121.630575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8121.636522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8122.750826] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8122.762802] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8122.803284] Lustre: lustre-MDT0000: new disk, initializing [ 8122.832582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8122.840063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8124.284441] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8128.504308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8128.549586] Lustre: 304604:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8128.574844] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8128.576594] Lustre: Skipped 1 previous similar message [ 8128.611467] Lustre: lustre-MDT0001: new disk, initializing [ 8128.645796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8128.658938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8128.662912] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8129.952437] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8133.008279] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8134.199471] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8136.600387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8136.731516] Lustre: lustre-OST0000: new disk, initializing [ 8136.733557] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8136.736212] Lustre: 306346:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8136.769088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8137.164070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8137.169191] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8137.217063] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8138.806895] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8142.037019] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8142.799868] Lustre: Mounted lustre-client [ 8143.135301] LustreError: 307098:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff8c32026be800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 8143.158318] Lustre: Unmounted lustre-client [ 8147.425627] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8147.425672] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8147.428794] LustreError: Skipped 1 previous similar message [ 8147.433204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8147.437748] Lustre: Skipped 1 previous similar message [ 8152.219720] Lustre: server umount lustre-OST0000 complete [ 8154.593142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8154.595644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8154.598895] Lustre: Skipped 1 previous similar message [ 8154.600658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8158.177266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8158.177807] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8159.583330] Lustre: server umount lustre-MDT0000 complete [ 8160.897198] LustreError: 305383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778930922 with bad export cookie 7079791909704568480 [ 8160.900484] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8160.900854] LustreError: 305383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8161.024563] Lustre: server umount lustre-MDT0001 complete [ 8164.469229] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 8165.431624] Key type lgssc unregistered [ 8200.416358] LNet: 308376:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8200.418961] LNetError: 308376:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8275.174362] LNet: Removed LNI 192.168.202.146@tcp [ 8275.488975] Key type .llcrypt unregistered [ 8275.489962] Key type ._llcrypt unregistered [ 8281.584138] Key type ._llcrypt registered [ 8281.586058] Key type .llcrypt registered [ 8281.628158] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 8288.114640] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 8288.465845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8288.520707] alg: No test for adler32 (adler32-zlib) [ 8289.370454] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 8289.450476] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 8291.032145] Key type lgssc registered [ 8291.369075] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8301.482460] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 07:31:03 (1778931063) [ 8303.708254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8303.713368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8304.798234] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8304.810842] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8304.849071] Lustre: lustre-MDT0000: new disk, initializing [ 8304.872631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8304.878778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8306.157430] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8310.298561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8310.340292] Lustre: 311541:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8310.362880] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8310.364553] Lustre: Skipped 1 previous similar message [ 8310.399908] Lustre: lustre-MDT0001: new disk, initializing [ 8310.431633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8310.442398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8310.445611] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8311.706967] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8314.771447] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8315.995515] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8318.256745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8318.359582] Lustre: lustre-OST0000: new disk, initializing [ 8318.361388] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8318.363914] Lustre: 313284:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8318.390890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8320.272307] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8323.462199] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8324.104477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8324.108038] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8324.141604] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8334.309850] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8334.310154] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8334.317750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8334.324551] Lustre: Skipped 1 previous similar message [ 8338.845912] Lustre: server umount lustre-OST0000 complete [ 8341.473088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8341.476655] LustreError: Skipped 1 previous similar message [ 8341.478252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8341.481331] Lustre: Skipped 1 previous similar message [ 8341.482865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8344.545109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8344.545720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8346.589456] Lustre: server umount lustre-MDT0000 complete [ 8347.818292] LustreError: 311534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778931109 with bad export cookie 12481784754868356403 [ 8347.821791] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8347.822577] LustreError: 311534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8347.945739] Lustre: server umount lustre-MDT0001 complete [ 8351.484434] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 8352.523864] Key type lgssc unregistered [ 8352.659360] LNet: 314910:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8352.661987] LNetError: 314910:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8352.668419] LNet: Removed LNI 192.168.202.146@tcp [ 8352.973796] Key type .llcrypt unregistered [ 8352.975334] Key type ._llcrypt unregistered [ 8362.506246] Key type ._llcrypt registered [ 8362.507348] Key type .llcrypt registered [ 8362.548890] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 8363.023539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8363.039049] alg: No test for adler32 (adler32-zlib) [ 8363.894023] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 8363.982238] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 8365.560193] Key type lgssc registered [ 8365.902168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8368.409275] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8368.414882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8369.566465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8370.879779] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8373.769337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8373.883122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8375.163720] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8376.936681] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8378.196681] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8380.538062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8380.662703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8382.628622] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8384.483537] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8397.650400] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8398.200511] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8399.841017] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8399.843539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8399.847469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8401.694130] Lustre: server umount lustre-MDT0000 complete [ 8402.937824] LustreError: 316270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778931165 with bad export cookie 2633046475806657326 [ 8402.942724] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8404.449752] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8404.450399] LustreError: 316286:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8404.453348] Lustre: Skipped 1 previous similar message [ 8404.457734] LustreError: 316286:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 8404.459330] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8404.462617] Lustre: Skipped 1 previous similar message [ 8405.098671] Lustre: server umount lustre-MDT0001 complete [ 8410.328227] Lustre: lustre-OST0000: Not available for connect from 192.168.202.46@tcp (stopping) [ 8412.447685] Lustre: server umount lustre-OST0000 complete [ 8415.227159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8416.562834] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8417.490362] Lustre: 320033:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.146@tcp [ 8418.197579] Lustre: server umount MGS complete [ 8420.952815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8421.098447] LustreError: 320603:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8421.122727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8422.446267] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8425.329239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8425.449694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8426.741696] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8430.066184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8431.999130] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8433.761061] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8434.282910] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8435.408578] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8441.021387] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 8445.922258] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8445.924736] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8445.929344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8445.931317] Lustre: Skipped 1 previous similar message [ 8448.275600] Lustre: server umount lustre-OST0000 complete [ 8451.041785] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8451.042043] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8451.045568] Lustre: Skipped 1 previous similar message [ 8451.050302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8451.052009] Lustre: Skipped 1 previous similar message [ 8455.705336] Lustre: server umount lustre-MDT0000 complete [ 8456.161385] LustreError: 320604:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8456.165872] LustreError: 320604:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 8456.986281] LustreError: 320584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778931219 with bad export cookie 2633046475806658152 [ 8456.990157] LustreError: 320584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8456.991787] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8457.113514] Lustre: server umount lustre-MDT0001 complete [ 8460.685176] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 8461.798791] Key type lgssc unregistered [ 8461.941318] LNet: 323777:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8461.943771] LNetError: 323777:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8461.953372] LNet: Removed LNI 192.168.202.146@tcp [ 8462.255304] Key type .llcrypt unregistered [ 8462.256983] Key type ._llcrypt unregistered [ 8468.467464] Key type ._llcrypt registered [ 8468.468491] Key type .llcrypt registered [ 8468.512448] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 8474.951264] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 8475.400864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8475.452453] alg: No test for adler32 (adler32-zlib) [ 8476.342315] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 8476.436751] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 8478.016191] Key type lgssc registered [ 8478.396838] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8490.388461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8490.393754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8491.487812] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8491.500026] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8491.540554] Lustre: lustre-MDT0000: new disk, initializing [ 8491.567381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8491.575405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8492.949022] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8497.687581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8497.734786] Lustre: 326875:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8497.765719] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8497.767641] Lustre: Skipped 1 previous similar message [ 8497.811160] Lustre: lustre-MDT0001: new disk, initializing [ 8497.853476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8497.868638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8497.873200] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8499.179338] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8502.442264] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8503.661594] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8505.978511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8506.076764] Lustre: lustre-OST0000: new disk, initializing [ 8506.078664] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8506.081732] Lustre: 328616:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8506.109698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8507.993298] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8511.027492] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8512.440561] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8514.557570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8514.560894] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8514.597876] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8515.601643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 8517.025243] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8517.118664] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8519.654687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8519.654823] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8519.665442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8519.669295] LustreError: Skipped 1 previous similar message [ 8523.874208] Lustre: server umount lustre-OST0000 complete [ 8528.865245] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8528.868194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8528.872289] Lustre: Skipped 1 previous similar message [ 8528.874703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8528.878352] Lustre: Skipped 1 previous similar message [ 8529.889881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8529.891156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8531.355194] Lustre: server umount lustre-MDT0000 complete [ 8532.610136] LustreError: 326868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778931294 with bad export cookie 17579050240585358117 [ 8532.613061] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8532.613380] LustreError: 326868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8532.732300] Lustre: server umount lustre-MDT0001 complete [ 8535.593369] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 07:34:57 (1778931297) [ 8540.831250] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 8543.961299] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 8555.880730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8555.969702] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8555.981064] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8556.020655] Lustre: lustre-MDT0000: new disk, initializing [ 8556.046122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8556.054342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8557.365488] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8561.419199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8561.458045] Lustre: 333205:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8561.479672] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8561.481377] Lustre: Skipped 1 previous similar message [ 8561.515380] Lustre: lustre-MDT0001: new disk, initializing [ 8561.544549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8561.555697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8561.558953] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8562.938284] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8566.063460] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8567.206449] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8569.652541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8569.749488] Lustre: lustre-OST0000: new disk, initializing [ 8569.751660] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8569.754389] Lustre: 334947:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8569.785482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8571.070614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8571.074128] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8571.118623] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8571.746057] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8574.958546] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8586.724487] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8586.725894] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8586.726950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8586.726957] Lustre: Skipped 1 previous similar message [ 8586.732660] LustreError: Skipped 1 previous similar message [ 8586.747697] Lustre: Skipped 1 previous similar message [ 8591.123708] Lustre: server umount lustre-OST0000 complete [ 8596.962394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8596.963730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8596.965750] Lustre: Skipped 1 previous similar message [ 8596.967470] Lustre: Skipped 1 previous similar message [ 8598.714417] Lustre: server umount lustre-MDT0000 complete [ 8600.083799] LustreError: 333200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778931362 with bad export cookie 17579050240585358985 [ 8600.088692] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8600.212065] Lustre: server umount lustre-MDT0001 complete [ 8604.060949] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 8605.428221] Key type lgssc unregistered [ 8605.557533] LNet: 336968:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8605.560422] LNetError: 336968:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8605.574483] LNet: Removed LNI 192.168.202.146@tcp [ 8605.927487] Key type .llcrypt unregistered [ 8605.929377] Key type ._llcrypt unregistered [ 8642.674727] Key type ._llcrypt registered [ 8642.677195] Key type .llcrypt registered [ 8642.731017] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 8642.841493] Key type .llcrypt unregistered [ 8642.842636] Key type ._llcrypt unregistered [ 8671.324165] Key type ._llcrypt registered [ 8671.325301] Key type .llcrypt registered [ 8671.363696] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 8671.466301] Key type .llcrypt unregistered [ 8671.467816] Key type ._llcrypt unregistered [ 8683.918735] Key type ._llcrypt registered [ 8683.919749] Key type .llcrypt registered [ 8683.953232] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 8684.045256] Key type .llcrypt unregistered [ 8684.046294] Key type ._llcrypt unregistered [ 8696.435318] Key type ._llcrypt registered [ 8696.437031] Key type .llcrypt registered [ 8696.477149] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 8696.588378] Key type .llcrypt unregistered [ 8696.589674] Key type ._llcrypt unregistered [ 8725.138817] Key type ._llcrypt registered [ 8725.140377] Key type .llcrypt registered [ 8725.189217] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 8725.310509] Key type .llcrypt unregistered [ 8725.311767] Key type ._llcrypt unregistered [ 8737.630152] Key type ._llcrypt registered [ 8737.631767] Key type .llcrypt registered [ 8737.689520] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 8737.847355] Key type .llcrypt unregistered [ 8737.850566] Key type ._llcrypt unregistered [ 8751.273770] Key type ._llcrypt registered [ 8751.275537] Key type .llcrypt registered [ 8751.335754] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 8751.473329] Key type .llcrypt unregistered [ 8751.475204] Key type ._llcrypt unregistered [ 8763.564313] Key type ._llcrypt registered [ 8763.565402] Key type .llcrypt registered [ 8763.604459] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 8763.698176] Key type .llcrypt unregistered [ 8763.699437] Key type ._llcrypt unregistered [ 8776.469833] Key type ._llcrypt registered [ 8776.470870] Key type .llcrypt registered [ 8776.515138] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 8776.639263] Key type .llcrypt unregistered [ 8776.640308] Key type ._llcrypt unregistered [ 8787.853177] Key type ._llcrypt registered [ 8787.854429] Key type .llcrypt registered [ 8787.893725] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 8787.996291] Key type .llcrypt unregistered [ 8787.997491] Key type ._llcrypt unregistered [ 8811.622091] Key type ._llcrypt registered [ 8811.622979] Key type .llcrypt registered [ 8811.660216] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 8811.767240] Key type .llcrypt unregistered [ 8811.769100] Key type ._llcrypt unregistered [ 8823.205424] Key type ._llcrypt registered [ 8823.206610] Key type .llcrypt registered [ 8823.239774] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 8823.332165] Key type .llcrypt unregistered [ 8823.333130] Key type ._llcrypt unregistered [ 8834.059229] Key type ._llcrypt registered [ 8834.060698] Key type .llcrypt registered [ 8834.106317] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 8834.211323] Key type .llcrypt unregistered [ 8834.212717] Key type ._llcrypt unregistered [ 8852.235298] Key type ._llcrypt registered [ 8852.237276] Key type .llcrypt registered [ 8852.273732] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 8858.256656] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 8858.643787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8858.659761] alg: No test for adler32 (adler32-zlib) [ 8859.546755] Lustre: Lustre: Build Version: 2.17.53_3_g0d061c0 [ 8859.645329] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 8861.232175] Key type lgssc registered [ 8861.675930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8876.823530] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8876.832359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8877.954273] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 8877.972246] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 8878.024188] Lustre: tst250fs-MDT0000: new disk, initializing [ 8878.062541] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8878.074242] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8879.726404] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8884.987836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8885.031395] Lustre: 356621:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8885.063912] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 8885.066060] Lustre: Skipped 1 previous similar message [ 8885.105300] Lustre: tst250fs-MDT0001: new disk, initializing [ 8885.143679] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8885.159643] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8885.166189] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8886.971278] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8889.392941] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8892.264015] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8892.395267] Lustre: tst250fs-OST0000: new disk, initializing [ 8892.398021] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 8892.402424] Lustre: 358218:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8892.443625] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8894.705502] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8896.077643] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8896.084884] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8896.162332] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8899.979267] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8900.081542] Lustre: tst250fs-OST0001: new disk, initializing [ 8900.084132] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 8900.088057] Lustre: 359074:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8900.128736] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8902.409660] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8904.748154] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8904.754544] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8904.804734] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8905.728541] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 8920.037499] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8920.038578] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 8920.042558] Lustre: Skipped 1 previous similar message [ 8925.154781] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 8925.158872] Lustre: Skipped 4 previous similar messages [ 8930.274684] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 8930.278600] Lustre: Skipped 4 previous similar messages [ 8932.320129] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8932.397200] Lustre: server umount tst250fs-MDT0000 complete [ 8933.655392] LustreError: 357399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778931695 with bad export cookie 17389551713026879174 [ 8933.658826] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8933.659468] LustreError: 357399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8933.779365] Lustre: server umount tst250fs-MDT0001 complete [ 8945.626620] Lustre: server umount tst250fs-OST0000 complete [ 8957.602410] Lustre: server umount tst250fs-OST0001 complete [ 8965.184259] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 8968.546904] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 8982.814924] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 07:42:24 (1778931744) [ 8983.467298] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 8990.810297] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 8993.908523] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 9006.211749] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8716 sec ========== 07:42:47 (1778931767) [ 9006.810962] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 07:42:48 (1778931768) === [ 9008.564634] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 07:42:50 (1778931770) === [ 9024.827759] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 9026.121770] Key type lgssc unregistered [ 9026.256442] LNet: 368083:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9026.259874] LNetError: 368083:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9026.272351] LNet: Removed LNI 192.168.202.146@tcp [ 9026.642124] Key type .llcrypt unregistered [ 9026.643597] Key type ._llcrypt unregistered