[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 511425702 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002389] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.006850] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.007023] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009152] LSM: Security Framework initializing [ 0.010051] Yama: becoming mindful. [ 0.011029] SELinux: Initializing. [ 0.012048] *** VALIDATE selinux *** [ 0.018628] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.022718] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.023138] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024102] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025122] *** VALIDATE tmpfs *** [ 0.027132] *** VALIDATE proc *** [ 0.028191] *** VALIDATE cgroup *** [ 0.029008] *** VALIDATE cgroup2 *** [ 0.030270] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.031148] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.033027] Spectre V2 : User space: Vulnerable [ 0.034007] Speculative Store Bypass: Vulnerable [ 0.037261] debug: unmapping init [mem 0xffffffffb1c59000-0xffffffffb1c60fff] [ 0.039886] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040625] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041020] ... version: 2 [ 0.042013] ... bit width: 48 [ 0.043011] ... generic registers: 4 [ 0.044013] ... value mask: 0000ffffffffffff [ 0.045014] ... max period: 00007fffffffffff [ 0.046014] ... fixed-purpose events: 3 [ 0.047022] ... event mask: 000000070000000f [ 0.048323] rcu: Hierarchical SRCU implementation. [ 0.050538] smp: Bringing up secondary CPUs ... [ 0.051649] x86: Booting SMP configuration: [ 0.052024] .... node #0, CPUs: #1 #2 #3 [ 0.055301] smp: Brought up 1 node, 4 CPUs [ 0.057014] smpboot: Max logical packages: 1 [ 0.058017] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.127070] node 0 deferred pages initialised in 66ms [ 0.130110] devtmpfs: initialized [ 0.131287] x86/mm: Memory block size: 128MB [ 0.134736] gcov: version magic: 0x41383552 [ 0.137191] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.140086] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.143351] pinctrl core: initialized pinctrl subsystem [ 0.145225] [ 0.145863] ************************************************************* [ 0.148013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.150070] ** ** [ 0.153014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.155012] ** ** [ 0.158014] ** This means that this kernel is built to expose internal ** [ 0.160011] ** IOMMU data structures, which may compromise security on ** [ 0.163014] ** your system. ** [ 0.165012] ** ** [ 0.167012] ** If you see this message and you are not debugging the ** [ 0.169015] ** kernel, report this immediately to your vendor! ** [ 0.171007] ** ** [ 0.173011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.175010] ************************************************************* [ 0.177785] NET: Registered protocol family 16 [ 0.179442] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.182062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.184050] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.188027] cpuidle: using governor menu [ 0.189730] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.191397] PCI: Using configuration type 1 for base access [ 0.193106] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.201131] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.204067] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.207402] cryptd: max_cpu_qlen set to 1000 [ 0.210271] ACPI: Added _OSI(Module Device) [ 0.211027] ACPI: Added _OSI(Processor Device) [ 0.212029] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.213013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.218455] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.223375] ACPI: Interpreter enabled [ 0.225073] ACPI: PM: (supports S0 S3 S4 S5) [ 0.226012] ACPI: Using IOAPIC for interrupt routing [ 0.228100] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.231436] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.242296] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.244040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.247028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.250216] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.254365] acpiphp: Slot [2] registered [ 0.257171] acpiphp: Slot [5] registered [ 0.259364] acpiphp: Slot [6] registered [ 0.261214] acpiphp: Slot [7] registered [ 0.263119] acpiphp: Slot [8] registered [ 0.264109] acpiphp: Slot [9] registered [ 0.266098] acpiphp: Slot [10] registered [ 0.267082] acpiphp: Slot [3] registered [ 0.268086] acpiphp: Slot [4] registered [ 0.269097] acpiphp: Slot [11] registered [ 0.270102] acpiphp: Slot [12] registered [ 0.272181] acpiphp: Slot [13] registered [ 0.274121] acpiphp: Slot [14] registered [ 0.275167] acpiphp: Slot [15] registered [ 0.277100] acpiphp: Slot [16] registered [ 0.278093] acpiphp: Slot [17] registered [ 0.279058] acpiphp: Slot [18] registered [ 0.280059] acpiphp: Slot [19] registered [ 0.281080] acpiphp: Slot [20] registered [ 0.283088] acpiphp: Slot [21] registered [ 0.284113] acpiphp: Slot [22] registered [ 0.286069] acpiphp: Slot [23] registered [ 0.287088] acpiphp: Slot [24] registered [ 0.288095] acpiphp: Slot [25] registered [ 0.289077] acpiphp: Slot [26] registered [ 0.290089] acpiphp: Slot [27] registered [ 0.291059] acpiphp: Slot [28] registered [ 0.292095] acpiphp: Slot [29] registered [ 0.294155] acpiphp: Slot [30] registered [ 0.295072] acpiphp: Slot [31] registered [ 0.296063] PCI host bridge to bus 0000:00 [ 0.298023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.300021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.302022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.305025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.307028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.310027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.312115] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.313640] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.315904] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.324016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.332016] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.334023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.336016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.338018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.340078] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.342753] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.345034] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.347326] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.352015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.364018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.367988] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.374420] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.380025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.388022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.413016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.428060] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.442019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.455023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.477015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.487856] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.503019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.518024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.548019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.558716] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.566017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.572016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.592017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.602621] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.608018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.618017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.631018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.640927] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.647019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.654017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.673044] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.687832] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.690373] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.692341] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.694426] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.697236] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.703027] iommu: Default domain type: Passthrough [ 0.704436] SCSI subsystem initialized [ 0.705120] ACPI: bus type USB registered [ 0.706141] usbcore: registered new interface driver usbfs [ 0.707095] usbcore: registered new interface driver hub [ 0.709194] usbcore: registered new device driver usb [ 0.711195] pps_core: LinuxPPS API ver. 1 registered [ 0.713008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.715048] PTP clock support registered [ 0.717085] EDAC MC: Ver: 3.0.0 [ 0.719128] PCI: Using ACPI for IRQ routing [ 0.720909] NetLabel: Initializing [ 0.722013] NetLabel: domain hash size = 128 [ 0.724012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.726083] NetLabel: unlabeled traffic allowed by default [ 0.728123] vgaarb: loaded [ 0.730337] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.732015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.737394] clocksource: Switched to clocksource kvm-clock [ 0.850912] VFS: Disk quotas dquot_6.6.0 [ 0.852693] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.855592] *** VALIDATE ramfs *** [ 0.856856] *** VALIDATE hugetlbfs *** [ 0.858389] pnp: PnP ACPI init [ 0.860721] pnp: PnP ACPI: found 6 devices [ 0.878028] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.881071] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.883076] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.885011] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.887021] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.888949] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.891331] NET: Registered protocol family 2 [ 0.893383] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.897905] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.901412] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.906641] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.911240] TCP: Hash tables configured (established 65536 bind 65536) [ 0.914414] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.917516] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.920310] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.923365] NET: Registered protocol family 1 [ 0.926213] RPC: Registered named UNIX socket transport module. [ 0.928491] RPC: Registered udp transport module. [ 0.930221] RPC: Registered tcp transport module. [ 0.931927] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.934379] NET: Registered protocol family 44 [ 0.935937] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.939393] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.941352] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.943914] PCI: CLS 0 bytes, default 64 [ 0.946268] Unpacking initramfs... [ 2.399880] debug: unmapping init [mem 0xffff8ea77cc54000-0xffff8ea77ffbffff] [ 2.406398] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.408531] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.412459] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.943357] Initialise system trusted keyrings [ 2.944729] Key type blacklist registered [ 2.946315] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.955164] zbud: loaded [ 2.957558] *** VALIDATE nfs *** [ 2.958472] *** VALIDATE nfs4 *** [ 2.959448] pstore: using deflate compression [ 2.963052] Platform Keyring initialized [ 3.049896] NET: Registered protocol family 38 [ 3.051720] Key type asymmetric registered [ 3.053569] Asymmetric key parser 'x509' registered [ 3.055572] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.058607] io scheduler mq-deadline registered [ 3.060211] io scheduler kyber registered [ 3.061844] io scheduler bfq registered [ 3.063784] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.066804] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.069752] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.072480] ACPI: Power Button [PWRF] [ 3.077558] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.084192] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.098691] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.107313] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.125706] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.154216] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.184184] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.189340] Non-volatile memory driver v1.3 [ 3.191351] Linux agpgart interface v0.103 [ 3.226604] virtio_blk virtio1: [vda] 134768 512-byte logical blocks (69.0 MB/65.8 MiB) [ 3.229909] vda: detected capacity change from 0 to 69001216 [ 3.254488] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.256797] vdb: detected capacity change from 0 to 1073741824 [ 3.272829] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.275891] vdc: detected capacity change from 0 to 2621440000 [ 3.295819] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.298828] vdd: detected capacity change from 0 to 2621440000 [ 3.314166] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.317043] vde: detected capacity change from 0 to 4294967296 [ 3.331969] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.334624] vdf: detected capacity change from 0 to 4294967296 [ 3.342642] libphy: Fixed MDIO Bus: probed [ 3.359388] usbcore: registered new interface driver usbserial_generic [ 3.362351] usbserial: USB Serial support registered for generic [ 3.365060] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.369591] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.381639] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.388639] mousedev: PS/2 mouse device common for all mice [ 3.394370] rtc_cmos 00:05: RTC can wake from S4 [ 3.396933] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.399524] rtc_cmos 00:05: registered as rtc0 [ 3.403610] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.403862] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.407819] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.409209] intel_pstate: CPU model not supported [ 3.415610] hid: raw HID events driver (C) Jiri Kosina [ 3.417817] usbcore: registered new interface driver usbhid [ 3.419970] usbhid: USB HID core driver [ 3.421571] drop_monitor: Initializing network drop monitor service [ 3.423983] Initializing XFRM netlink socket [ 3.428925] NET: Registered protocol family 10 [ 3.433383] Segment Routing with IPv6 [ 3.435072] NET: Registered protocol family 17 [ 3.438133] mpls_gso: MPLS GSO support [ 3.444842] RAS: Correctable Errors collector initialized. [ 3.447114] AVX version of gcm_enc/dec engaged. [ 3.448909] AES CTR mode by8 optimization enabled [ 3.537699] sched_clock: Marking stable (3537675674, 0)->(4406418504, -868742830) [ 3.541529] registered taskstats version 1 [ 3.543198] Loading compiled-in X.509 certificates [ 3.544853] zswap: loaded using pool lzo/zbud [ 3.571777] Key type big_key registered [ 3.583356] Key type encrypted registered [ 3.585182] ima: No TPM chip found, activating TPM-bypass! [ 3.586560] ima: Allocated hash algorithm: sha1 [ 3.587838] ima: No architecture policies found [ 3.589203] evm: Initialising EVM extended attributes: [ 3.591115] evm: security.selinux [ 3.592231] evm: security.ima [ 3.593374] evm: security.capability [ 3.594658] evm: HMAC attrs: 0x1 [ 3.597228] rtc_cmos 00:05: setting system clock to 2026-05-25 08:42:31 UTC (1779698551) [ 3.602771] debug: unmapping init [mem 0xffffffffb2c03000-0xffffffffb2dfffff] [ 3.605978] debug: unmapping init [mem 0xffffffffb1982000-0xffffffffb1c58fff] [ 3.614138] Write protecting the kernel read-only data: 28672k [ 3.617094] debug: unmapping init [mem 0xffffffffb0003000-0xffffffffb01fffff] [ 3.618660] debug: unmapping init [mem 0xffffffffb0914000-0xffffffffb09fffff] [ 3.653947] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.662814] systemd[1]: Detected virtualization kvm. [ 3.664784] systemd[1]: Detected architecture x86-64. [ 3.666648] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.696353] systemd[1]: No hostname configured. [ 3.698069] systemd[1]: Set hostname to . [ 3.700289] random: systemd: uninitialized urandom read (16 bytes read) [ 3.702253] systemd[1]: Initializing machine ID from random generator. [ 3.754629] random: ln: uninitialized urandom read (6 bytes read) [ 3.839423] random: systemd: uninitialized urandom read (16 bytes read) [ 3.842855] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.849153] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.855842] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.446494] device-mapper: uevent: version 1.0.3 [ 4.448261] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.164924] random: fast init done [ 5.172477] virtio_net virtio0 ens2: renamed from eth0 [ 5.435591] scsi host0: ata_piix [ 5.452926] scsi host1: ata_piix [ 5.454760] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.457368] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.018163] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.105313] random: crng init done [ 10.106616] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.547292] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.687977] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.931021] SELinux: Disabled at runtime. [ 11.993873] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.002098] systemd[1]: Detected virtualization kvm. [ 12.003679] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.525621] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.528758] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.533349] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.538509] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.542191] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.549085] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.553815] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ 12.645713] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.995001] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.320683] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.340549] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.485475] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.507150] EDAC sbridge: Ver: 1.1.2 [ 15.257967] Key type dns_resolver registered [ 15.564535] NFS: Registering the id_resolver key type [ 15.566590] Key type id_resolver registered [ 15.568478] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg205-server login: [ 41.100863] libcfs: loading out-of-tree module taints kernel. [ 41.114201] Key type ._llcrypt registered [ 41.115486] Key type .llcrypt registered [ 41.167329] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 49.485873] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 50.196979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 50.205416] alg: No test for adler32 (adler32-zlib) [ 51.214988] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 51.549531] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 53.183207] Key type lgssc registered [ 53.877713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 64.204641] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 79.922278] hrtimer: interrupt took 3522051 ns [ 111.079312] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 126.102586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 126.132781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 127.417857] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 127.476978] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 127.584866] Lustre: lustre-MDT0000: new disk, initializing [ 127.690256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 127.708916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 132.669708] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 148.964908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 149.139223] Lustre: 6475:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 149.164715] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 149.168233] Lustre: Skipped 1 previous similar message [ 149.239091] Lustre: lustre-MDT0001: new disk, initializing [ 149.310530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 149.339453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 149.353350] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 154.984644] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 160.957631] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 172.410686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 172.751270] Lustre: lustre-OST0000: new disk, initializing [ 172.754391] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 172.771755] Lustre: 8378:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 172.811323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 179.252889] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.748363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 179.763877] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 179.813718] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 196.339835] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 196.563556] Lustre: lustre-OST0001: new disk, initializing [ 196.574625] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 196.586761] Lustre: 9433:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 196.743550] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 203.972221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 203.998525] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 204.244495] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 207.094404] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.815787] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 230.281608] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 238.459710] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing check_logdir /tmp/testlogs/ [ 244.137093] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing yml_node [ 248.521950] Lustre: DEBUG MARKER: Client: 2.17.53.24 [ 251.463200] Lustre: DEBUG MARKER: MDS: 2.17.53.24 [ 254.226673] Lustre: DEBUG MARKER: OSS: 2.17.53.24 [ 256.109201] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon May 25 04:46:41 EDT 2026 [ 276.598219] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 278.338259] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 296.418628] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 296.421048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 296.437841] Lustre: Skipped 1 previous similar message [ 296.447417] Lustre: Skipped 2 previous similar messages [ 300.929593] Lustre: server umount lustre-MDT0000 complete [ 305.414088] LustreError: 6468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779698853 with bad export cookie 6081520876681932145 [ 305.415323] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 305.445802] LustreError: 6468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 306.656828] LustreError: 9441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 306.657403] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 306.657877] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 306.704821] LustreError: 9441:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 306.735326] Lustre: Skipped 3 previous similar messages [ 310.688327] LustreError: 6483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 310.688620] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 310.731870] LustreError: 6483:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 310.776739] Lustre: Skipped 2 previous similar messages [ 312.066514] Lustre: server umount lustre-MDT0001 complete [ 317.657881] Lustre: server umount lustre-OST0000 complete [ 323.766230] Lustre: server umount lustre-OST0001 complete [ 333.049101] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 342.811541] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 387.125232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 387.504968] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 387.585212] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 387.741448] Lustre: lustre-MDT0000: new disk, initializing [ 387.831616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 387.846363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 392.741978] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 405.107212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 405.290858] Lustre: 14957:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 405.351935] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 405.355175] Lustre: Skipped 1 previous similar message [ 405.479317] Lustre: lustre-MDT0001: new disk, initializing [ 405.643285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 405.687603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 405.702338] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 411.431539] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 420.274606] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 425.719401] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 433.458108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 433.965661] Lustre: lustre-OST0000: new disk, initializing [ 433.968928] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 433.974334] Lustre: 16698:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 434.061810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 435.852968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 435.870770] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 435.980113] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 441.192123] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 449.608216] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 455.590527] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 456.004672] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 460.850982] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 461.050432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 466.912355] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 466.913854] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 466.931943] LustreError: Skipped 1 previous similar message [ 466.932930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 466.993543] Lustre: Skipped 1 previous similar message [ 469.347527] Lustre: server umount lustre-OST0000 complete [ 475.106991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 475.120447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 475.129948] Lustre: Skipped 1 previous similar message [ 475.174764] Lustre: Skipped 2 previous similar messages [ 480.455746] Lustre: server umount lustre-MDT0000 complete [ 484.321095] LustreError: 14968:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 484.348739] LustreError: 14968:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 484.901844] LustreError: 14947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779699032 with bad export cookie 6081520876681933678 [ 484.910693] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 484.916962] LustreError: 14947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 485.573419] Lustre: server umount lustre-MDT0001 complete [ 494.451379] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 04:50:39 (1779699039) [ 517.221988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 518.103527] LustreError: 19426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 518.232796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 523.232987] LustreError: 19427:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 523.583579] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 533.476666] LustreError: 19426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 533.510932] LustreError: 19426:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 535.184201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 541.373291] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 548.928984] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 554.314919] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 557.090664] Lustre: Failing over lustre-MDT0000 [ 557.441392] Lustre: server umount lustre-MDT0000 complete [ 560.166535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 560.171166] LustreError: 19422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 560.180134] Lustre: Skipped 1 previous similar message [ 560.230415] LustreError: 19422:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 576.479853] Lustre: 3629:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779699108/real 1779699108] req@ffff8ea6c3089880 x1866149238313728/t0(0) o400->MGC192.168.202.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1779699124 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 576.522763] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 579.065442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 587.777330] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x5465e7041cbfb8f4 [ 587.800039] Lustre: MGC192.168.202.105@tcp: Connection restored to 0@lo (at 0@lo) [ 588.194160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 588.200502] Lustre: Skipped 1 previous similar message [ 593.067380] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 593.378931] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 593.389138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 593.434735] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 602.321184] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 607.414559] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 607.470853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 607.473556] Lustre: Skipped 2 previous similar messages [ 614.034456] Lustre: server umount lustre-MDT0000 complete [ 617.953452] LustreError: 20137:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 617.984985] LustreError: 20137:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 618.816333] LustreError: 21749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779699166 with bad export cookie 6081520876681935092 [ 618.824871] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 618.839858] LustreError: 21749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 619.377918] Lustre: server umount lustre-MDT0001 complete [ 630.879946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 637.427974] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 648.083864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 653.551188] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 660.229602] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 664.822534] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 667.204699] Lustre: Failing over lustre-MDT0000 [ 667.628897] Lustre: server umount lustre-MDT0000 complete [ 669.152400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 669.164136] Lustre: Skipped 1 previous similar message [ 669.168462] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 683.488362] LustreError: 23540:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 683.513926] LustreError: 23540:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 685.536220] Lustre: 3626:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779699217/real 1779699217] req@ffff8ea7e6edd880 x1866149238344576/t0(0) o400->MGC192.168.202.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1779699233 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 685.567643] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 688.840895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 695.778647] LustreError: 3625:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8ea6c2a99c00 x1866149238347520/t0(0) o250->MGC192.168.202.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 696.191927] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 701.411834] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 701.426257] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 701.430040] Lustre: Skipped 1 previous similar message [ 701.449091] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 701.556828] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 713.248461] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 719.010151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 719.011630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 719.015325] Lustre: Skipped 1 previous similar message [ 719.018969] Lustre: Skipped 2 previous similar messages [ 724.306335] Lustre: server umount lustre-MDT0000 complete [ 729.088742] LustreError: 24752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779699276 with bad export cookie 6081520876681936030 [ 729.090476] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 729.104582] LustreError: 24752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 729.786638] Lustre: server umount lustre-MDT0001 complete [ 741.085977] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 04:54:46 (1779699286) [ 762.252754] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 774.084814] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 818.033441] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 827.819448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 828.035029] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 828.062230] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 828.178579] Lustre: lustre-MDT0000: new disk, initializing [ 828.257358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 828.264710] Lustre: Skipped 3 previous similar messages [ 828.285222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 832.919398] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 843.985784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 844.169167] Lustre: 28417:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 844.586675] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 849.556803] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 857.777988] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 863.021785] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 872.254811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 872.532793] Lustre: 30159:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 873.884394] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 873.949254] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 874.006564] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 874.018689] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 879.408274] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 888.088823] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 894.991339] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 920.032357] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 920.034499] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 920.040943] LustreError: Skipped 1 previous similar message [ 920.044616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 920.095699] Lustre: Skipped 1 previous similar message [ 923.753636] Lustre: server umount lustre-OST0000 complete [ 934.107196] Lustre: server umount lustre-MDT0000 complete [ 935.398565] LustreError: 28427:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 935.413146] LustreError: 28427:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 938.477926] LustreError: 28406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779699486 with bad export cookie 6081520876681936429 [ 938.481123] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 938.487239] LustreError: 28406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 938.910410] Lustre: server umount lustre-MDT0001 complete [ 954.751453] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 959.157594] Key type lgssc unregistered [ 959.406726] LNet: 32014:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 959.414357] LNetError: 32014:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 959.426057] LNet: Removed LNI 192.168.202.105@tcp [ 960.492172] Key type .llcrypt unregistered [ 960.496944] Key type ._llcrypt unregistered [ 990.876503] Key type ._llcrypt registered [ 990.879961] Key type .llcrypt registered [ 990.995143] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 992.365553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 992.444095] alg: No test for adler32 (adler32-zlib) [ 993.543791] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 993.761714] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 995.431185] Key type lgssc registered [ 996.686473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1004.792477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1004.827115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1006.432693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1011.179479] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1016.810186] LustreError: 33420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1016.840500] LustreError: 33420:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1021.297187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1021.530757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1026.488880] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1032.405794] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1036.708272] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1044.755234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1045.166728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1046.193329] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1051.387483] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1052.548064] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1057.984250] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1073.611880] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 05:00:19 (1779699619) [ 1075.880181] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1078.408758] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1086.112904] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 05:00:32 (1779699632) [ 1095.676633] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 05:00:41 (1779699641) [ 1096.761964] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1107.363477] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1107.373251] Lustre: Skipped 2 previous similar messages [ 1109.335378] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 05:00:55 (1779699655) [ 1112.604337] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1112.608529] Lustre: Skipped 3 previous similar messages [ 1130.627575] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1130.633553] Lustre: Skipped 6 previous similar messages [ 1133.685267] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 05:01:19 (1779699679) [ 1136.330274] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1139.685722] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1147.914353] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 05:01:34 (1779699694) [ 1149.800884] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1149.805786] Lustre: Skipped 3 previous similar messages [ 1162.291649] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 05:01:48 (1779699708) [ 1179.620714] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1179.628147] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1179.634600] LustreError: Skipped 1 previous similar message [ 1179.636846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1179.660413] Lustre: Skipped 1 previous similar message [ 1179.690429] Lustre: Skipped 1 previous similar message [ 1184.737535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1184.745341] Lustre: Skipped 1 previous similar message [ 1189.864488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1189.883061] Lustre: Skipped 1 previous similar message [ 1190.371391] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1190.650405] Lustre: server umount lustre-OST0000 complete [ 1200.101232] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1200.119047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1200.127356] Lustre: Skipped 1 previous similar message [ 1202.796076] Lustre: server umount lustre-MDT0000 complete [ 1205.218194] LustreError: 35524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1205.236793] LustreError: 35524:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1206.844560] LustreError: 33401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779699754 with bad export cookie 16269025449990852083 [ 1206.853532] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1206.858556] LustreError: 33401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1207.511332] Lustre: server umount lustre-MDT0001 complete [ 1220.644718] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 1223.507600] Key type lgssc unregistered [ 1223.858404] LNet: 40933:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1223.868911] LNetError: 40933:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1223.888241] LNet: Removed LNI 192.168.202.105@tcp [ 1224.937133] Key type .llcrypt unregistered [ 1224.940442] Key type ._llcrypt unregistered [ 1243.711833] Key type ._llcrypt registered [ 1243.715439] Key type .llcrypt registered [ 1243.846630] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 1260.628602] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 1262.114518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1262.151247] alg: No test for adler32 (adler32-zlib) [ 1263.368710] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 1263.729383] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 1265.423532] Key type lgssc registered [ 1267.236897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1308.674973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1308.708329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1310.110359] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1310.131854] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1310.230683] Lustre: lustre-MDT0000: new disk, initializing [ 1310.330557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1310.346637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1315.787790] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1327.094222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1327.221946] Lustre: 44056:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1327.291717] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1327.300517] Lustre: Skipped 1 previous similar message [ 1327.377755] Lustre: lustre-MDT0001: new disk, initializing [ 1327.507964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1327.564851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1327.580900] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1332.264769] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1341.249365] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1346.008744] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1353.664313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1353.870307] Lustre: lustre-OST0000: new disk, initializing [ 1353.873729] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1353.878822] Lustre: 45800:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1353.943081] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1361.567317] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1362.573905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1362.589967] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1362.757737] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1371.827593] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1377.386456] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1377.613164] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1382.910884] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1383.318204] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1388.513656] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1388.516169] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1388.520435] LustreError: Skipped 1 previous similar message [ 1388.531531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1388.543676] Lustre: Skipped 1 previous similar message [ 1391.819815] Lustre: server umount lustre-OST0000 complete [ 1398.754442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1398.780502] Lustre: Skipped 1 previous similar message [ 1398.785708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1398.790021] Lustre: Skipped 1 previous similar message [ 1402.721891] Lustre: server umount lustre-MDT0000 complete [ 1403.872849] LustreError: 44063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1403.881803] LustreError: 44063:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1407.854753] LustreError: 44049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779699955 with bad export cookie 13653931287221391556 [ 1407.858478] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1407.864884] LustreError: 44049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1408.240815] Lustre: server umount lustre-MDT0001 complete [ 1419.730441] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 05:06:05 (1779699965) [ 1427.352539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1427.654445] LustreError: 48113:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1427.690263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1431.785462] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1433.073243] LustreError: 48114:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1437.200432] LustreError: 48113:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1441.173287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1441.526247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1446.607561] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1452.887279] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1457.186520] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1464.873452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1465.287805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1471.614155] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1477.637758] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1482.252173] LustreError: 50302:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1483.208713] Lustre: Modifying parameter general.timeout=1 in log params [ 1484.735913] Lustre: Modifying parameter general.timeout=3 in log params [ 1484.743276] Lustre: Skipped 1 previous similar message [ 1487.471545] Lustre: Modifying parameter general.timeout=6 in log params [ 1487.475544] Lustre: Skipped 2 previous similar messages [ 1492.458077] Lustre: Modifying parameter general.timeout=11 in log params [ 1492.464301] Lustre: Skipped 4 previous similar messages [ 1501.234049] Lustre: Modifying parameter general.timeout=19 in log params [ 1501.237022] Lustre: Skipped 7 previous similar messages [ 1517.288526] Lustre: Modifying parameter general.timeout=34 in log params [ 1517.291064] Lustre: Skipped 14 previous similar messages [ 1550.044345] Lustre: Modifying parameter general.timeout=67 in log params [ 1550.047531] Lustre: Skipped 32 previous similar messages [ 1617.718967] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 05:09:23 (1779700163) [ 1619.125065] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1619.137146] Lustre: Skipped 63 previous similar messages [ 1621.709855] Lustre: 57257:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1623.796948] Lustre: 57356:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1625.930699] Lustre: 57468:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1632.261288] Lustre: 57762:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1632.276069] Lustre: 57762:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 2 previous similar messages [ 1665.158969] Lustre: 59148:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1665.194324] LustreError: 59148:0:(mgs_handler.c:1202:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1669.784474] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 05:10:15 (1779700215) [ 1687.520496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1687.531602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1687.532943] Lustre: Skipped 2 previous similar messages [ 1687.552584] Lustre: Skipped 2 previous similar messages [ 1689.299936] Lustre: server umount lustre-MDT0000 complete [ 1692.646506] LustreError: 48824:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1692.662377] LustreError: 48824:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1693.211581] LustreError: 48094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779700241 with bad export cookie 13653931287221392438 [ 1693.215588] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1693.224142] LustreError: 48094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1693.561189] Lustre: server umount lustre-MDT0001 complete [ 1706.515397] Lustre: server umount lustre-OST0000 complete [ 1732.182158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1732.436806] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1732.748125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1737.321731] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1746.687810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1746.804768] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1746.819568] Lustre: 61678:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1751.310580] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1756.522272] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1761.533410] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1768.518746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1768.646612] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1768.920272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1768.924339] Lustre: Skipped 1 previous similar message [ 1773.979438] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1779.390293] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1780.975716] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1780.981393] Lustre: Skipped 22 previous similar messages [ 1781.022414] Lustre: 63906:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1788.387636] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1788.391950] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1788.399224] Lustre: Skipped 1 previous similar message [ 1788.414090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1793.509279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1793.515560] Lustre: Skipped 1 previous similar message [ 1794.036593] Lustre: server umount lustre-OST0000 complete [ 1798.119173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1798.133398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1798.147054] Lustre: Skipped 1 previous similar message [ 1803.752166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1803.759051] Lustre: Skipped 4 previous similar messages [ 1804.139625] Lustre: server umount lustre-MDT0000 complete [ 1807.689262] LustreError: 61673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779700355 with bad export cookie 13653931287221395588 [ 1807.691490] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1807.696929] LustreError: 61673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1808.105266] Lustre: server umount lustre-MDT0001 complete [ 1819.567675] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 1822.484634] Key type lgssc unregistered [ 1822.842658] LNet: 65098:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1822.853692] LNetError: 65098:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1822.874587] LNet: Removed LNI 192.168.202.105@tcp [ 1823.810740] Key type .llcrypt unregistered [ 1823.813498] Key type ._llcrypt unregistered [ 1849.265941] Key type ._llcrypt registered [ 1849.269090] Key type .llcrypt registered [ 1849.369666] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 1850.609839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1850.681836] alg: No test for adler32 (adler32-zlib) [ 1851.744798] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 1851.992491] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 1853.735180] Key type lgssc registered [ 1855.105984] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1863.223384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1863.248269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1864.747884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1869.008367] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1874.923045] LustreError: 66494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1874.944650] LustreError: 66494:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1878.204242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1878.794832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1883.796401] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1890.302457] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1893.930252] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1900.815139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1901.437273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1907.119909] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1912.195515] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1920.412776] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1929.700397] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1929.703997] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1929.710524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1930.220308] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1930.237832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1930.237841] Lustre: Skipped 1 previous similar message [ 1930.255302] Lustre: Skipped 1 previous similar message [ 1935.329677] LustreError: 67209:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1935.369228] LustreError: 67209:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1935.405467] Lustre: server umount lustre-MDT0000 complete [ 1938.784596] LustreError: 66476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779700486 with bad export cookie 4470877899882918053 [ 1938.788921] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1938.792141] LustreError: 66476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1939.004826] Lustre: server umount lustre-MDT0001 complete [ 1953.099899] Lustre: server umount lustre-OST0000 complete [ 1983.405481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1983.682542] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1983.734726] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1984.071366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1988.287141] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1997.539733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1997.616039] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1997.687267] Lustre: 71029:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1997.783280] LustreError: 71040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1998.132350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2003.403773] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2009.574714] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2013.798814] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2022.034688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2022.339871] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2022.935469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2030.678665] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2037.467780] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2038.889371] Lustre: 73266:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2040.937465] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2047.469870] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2047.470284] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2047.471318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2047.471324] Lustre: Skipped 1 previous similar message [ 2047.528824] LustreError: Skipped 1 previous similar message [ 2052.584336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2052.595505] Lustre: Skipped 2 previous similar messages [ 2052.987979] Lustre: server umount lustre-OST0000 complete [ 2057.699420] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2057.709929] Lustre: Skipped 1 previous similar message [ 2057.714359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2057.727152] Lustre: Skipped 1 previous similar message [ 2064.232986] Lustre: server umount lustre-MDT0000 complete [ 2067.937842] LustreError: 71773:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2067.990826] LustreError: 71773:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2070.377760] LustreError: 72685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779700618 with bad export cookie 4470877899882918893 [ 2070.395598] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2070.871706] Lustre: server umount lustre-MDT0001 complete [ 2086.253737] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 2091.506715] Key type lgssc unregistered [ 2092.000614] LNet: 74463:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2092.026551] LNetError: 74463:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2092.058585] LNet: Removed LNI 192.168.202.105@tcp [ 2092.891437] Key type .llcrypt unregistered [ 2092.894441] Key type ._llcrypt unregistered [ 2122.273601] Key type ._llcrypt registered [ 2122.276485] Key type .llcrypt registered [ 2122.366154] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 2123.332064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2123.500201] alg: No test for adler32 (adler32-zlib) [ 2124.689564] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 2125.022282] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 2126.792177] Key type lgssc registered [ 2128.512604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2136.834087] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2136.859585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2138.576165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2143.610963] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2148.836137] LustreError: 75875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2148.867327] LustreError: 75875:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2153.905523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2153.961161] LustreError: 75876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2154.329072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2159.192495] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2165.017596] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2169.496508] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2177.352261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2177.851232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2184.304489] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2190.647277] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2196.864354] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2197.914173] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2199.848958] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2199.854846] Lustre: Skipped 1 previous similar message [ 2201.982798] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2201.995813] Lustre: Skipped 1 previous similar message [ 2205.999837] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2206.004343] Lustre: Skipped 3 previous similar messages [ 2214.972325] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2214.977237] Lustre: Skipped 9 previous similar messages [ 2231.093578] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2231.096650] Lustre: Skipped 14 previous similar messages [ 2263.823805] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2263.832637] Lustre: Skipped 34 previous similar messages [ 2300.297474] Lustre: 83640:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2303.307075] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 05:20:48 (1779700848) [ 2305.086992] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2307.852465] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 05:20:53 (1779700853) [ 2313.193199] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2313.199841] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2313.216432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2317.283631] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2317.290251] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2317.293119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2317.321099] Lustre: Skipped 1 previous similar message [ 2318.869522] Lustre: server umount lustre-OST0000 complete [ 2323.429813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2323.447132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2323.479924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2327.522196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2327.530451] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2329.818962] Lustre: server umount lustre-MDT0000 complete [ 2332.642668] LustreError: 78294:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2332.665638] LustreError: 78294:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2333.997437] LustreError: 75857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779700881 with bad export cookie 394729054889865576 [ 2334.007642] LustreError: 75857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2334.011843] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2334.406869] Lustre: server umount lustre-MDT0001 complete [ 2346.496384] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 2349.494483] Key type lgssc unregistered [ 2349.752560] LNet: 84877:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2349.758331] LNetError: 84877:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2350.824681] LNet: Removed LNI 192.168.202.105@tcp [ 2351.564198] Key type .llcrypt unregistered [ 2351.566282] Key type ._llcrypt unregistered [ 2355.817386] Key type ._llcrypt registered [ 2355.818800] Key type .llcrypt registered [ 2355.913530] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2360.078851] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules [ 2361.036574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2361.075721] alg: No test for adler32 (adler32-zlib) [ 2362.179356] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 2362.322104] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 2364.016944] Key type lgssc registered [ 2364.992932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2366.040076] LustreError: 86045:0:(lu_object.c:1479:lu_context_key_register()) cfs_race id 60d sleeping [ 2384.709880] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 2388.957032] LustreError: 86045:0:(lu_object.c:1479:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2396.156237] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2396.190265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2398.028045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2402.682973] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2407.407156] LustreError: 87051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2407.440707] LustreError: 87051:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2412.517055] LustreError: 87050:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2412.950788] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 05:22:38 (1779700958) [ 2417.641444] LustreError: 87051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2418.018823] LustreError: 87083:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2418.028288] LustreError: 87083:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2418.037498] LustreError: 87083:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2418.323846] Lustre: server umount lustre-MDT0000 complete [ 2430.448049] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 2432.767803] Key type lgssc unregistered [ 2433.076302] LNet: 88542:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2433.089124] LNetError: 88542:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2434.150771] LNet: Removed LNI 192.168.202.105@tcp [ 2434.932198] Key type .llcrypt unregistered [ 2434.936388] Key type ._llcrypt unregistered [ 2453.662610] Key type ._llcrypt registered [ 2453.665648] Key type .llcrypt registered [ 2453.809142] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 2455.150076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2455.234707] alg: No test for adler32 (adler32-zlib) [ 2456.426792] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 2456.637422] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 2458.279236] Key type lgssc registered [ 2459.379412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2465.741984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2465.757585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2467.214321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2470.801627] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2476.513785] LustreError: 89745:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2476.540976] LustreError: 89745:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2478.272815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2478.876112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2482.940799] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2488.101047] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2491.625425] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2497.284676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2497.706210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2502.543438] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2508.208556] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2560.078920] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 05:25:05 (1779701105) [ 2562.001465] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2563.830899] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 05:25:09 (1779701109) [ 2568.673605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2568.681815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2568.689061] Lustre: Skipped 2 previous similar messages [ 2568.701994] Lustre: Skipped 1 previous similar message [ 2573.723810] Lustre: server umount lustre-MDT0000 complete [ 2573.792628] LustreError: 90458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2573.807139] LustreError: 90458:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2577.336039] LustreError: 89727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779701125 with bad export cookie 10218487268151712550 [ 2577.337591] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2577.343173] LustreError: 89727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2577.648846] Lustre: server umount lustre-MDT0001 complete [ 2591.994612] Lustre: server umount lustre-OST0000 complete [ 2603.326802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2604.125357] LustreError: 93678:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2604.346088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2608.634588] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2609.635784] LustreError: 93679:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2614.754191] LustreError: 93679:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2617.687721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2618.236162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2623.142139] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2628.766071] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2632.172602] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2645.173100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2645.377814] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2645.387731] LustreError: 93667:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2645.413708] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2645.422380] LustreError: 95536:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2645.433321] LustreError: 95536:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2645.440153] LustreError: 95536:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2645.508851] Lustre: server umount lustre-OST0000 complete [ 2645.511680] LustreError: 95536:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2654.915351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2655.113934] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2655.120496] LustreError: 93665:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2655.147550] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2655.154814] LustreError: 96002:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2655.159268] LustreError: 96002:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2655.168741] LustreError: 96002:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2655.227433] Lustre: server umount lustre-OST0000 complete [ 2655.230552] LustreError: 96002:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2664.019353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2664.220892] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2664.227824] Lustre: Found index 0 for lustre-OST0000, updating log [ 2664.232707] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2664.286424] Lustre: lustre-OST0000: new disk, initializing [ 2664.289637] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2664.295353] Lustre: 96536:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2664.516392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2669.897979] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2673.749329] Lustre: Failing over lustre-OST0000 [ 2673.917047] Lustre: server umount lustre-OST0000 complete [ 2679.265526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2679.275359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2679.299480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2680.288447] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2680.308792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2680.311235] Lustre: Skipped 1 previous similar message [ 2683.411207] Lustre: server umount lustre-MDT0000 complete [ 2685.412618] LustreError: 93679:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2685.431310] LustreError: 93679:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2686.586522] LustreError: 93659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779701234 with bad export cookie 10218487268151715294 [ 2686.590419] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2686.593895] LustreError: 93659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2686.767432] Lustre: server umount lustre-MDT0001 complete [ 2695.534566] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 05:27:21 (1779701241) [ 2717.709441] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 2728.711570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2729.205803] LustreError: 98867:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2729.319179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2733.561412] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2741.814752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2746.272440] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2748.870350] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2754.964986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2761.433926] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2770.462543] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2770.566735] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2770.657348] Lustre: lustre-OST0001: new disk, initializing [ 2770.668341] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2770.678893] Lustre: 101134:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2770.920233] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2770.923739] Lustre: Skipped 2 previous similar messages [ 2772.862274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2772.875775] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2772.938409] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2776.669292] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2785.521387] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2793.346446] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2798.565939] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2798.576045] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2798.603553] Lustre: Skipped 1 previous similar message [ 2798.620432] Lustre: Skipped 2 previous similar messages [ 2803.687114] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2803.691263] Lustre: Skipped 3 previous similar messages [ 2812.383173] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2812.629448] Lustre: server umount lustre-MDT0001 complete [ 2813.929586] LustreError: 99445:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2813.963483] LustreError: 99445:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2822.977101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2823.091341] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2823.096058] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2823.125131] Lustre: 98856:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2824.638093] LustreError: 102671:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2824.643574] LustreError: 102671:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2824.657574] LustreError: 98859:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2824.788094] LustreError: 102686:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.105@tcp: cfg command failed: rc = -17 [ 2824.794588] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2824.794588] [ 2828.795966] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2829.226682] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2839.044045] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 05:29:44 (1779701384) [ 2855.454564] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 2874.204148] Lustre: 104701:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2896.263499] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2900.557234] Lustre: 105836:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2987.491468] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2987.492309] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2987.496777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2987.496787] Lustre: Skipped 5 previous similar messages [ 2987.508821] Lustre: Skipped 2 previous similar messages [ 2991.190754] Lustre: server umount lustre-MDT0000 complete [ 2992.611630] LustreError: 106179:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2992.643293] LustreError: 106179:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2994.957360] LustreError: 98848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779701542 with bad export cookie 10218487268151716288 [ 2994.970567] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2994.976763] LustreError: 98848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2995.300766] Lustre: server umount lustre-MDT0001 complete [ 3009.900192] Lustre: server umount lustre-OST0000 complete [ 3023.341375] Lustre: server umount lustre-OST0001 complete [ 3029.433389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3040.214248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3049.578802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3060.193603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3081.140919] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 3091.853077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3091.876860] Lustre: lustre-MDT0000: reset Object Index mappings [ 3092.638807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3092.645806] Lustre: Skipped 1 previous similar message [ 3097.243217] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3106.209195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3106.235377] Lustre: lustre-MDT0001: reset Object Index mappings [ 3113.066809] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3116.566347] Lustre: 110888:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3124.363850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3131.403603] LustreError: 111242:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3131.424645] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3131.431339] LustreError: 111242:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3133.501345] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3137.527062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:350 to 0x280000401:386) [ 3144.011377] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3149.834862] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 3149.846530] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3151.603838] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3160.368222] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3164.657325] Lustre: 112732:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3216.620264] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 05:36:02 (1779701762) [ 3245.030207] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3245.036515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3245.049333] Lustre: Skipped 1 previous similar message [ 3245.059592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3245.064486] Lustre: Skipped 3 previous similar messages [ 3250.100885] Lustre: server umount lustre-MDT0000 complete [ 3254.450695] LustreError: 109753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779701802 with bad export cookie 10218487268151763160 [ 3254.455827] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3254.460086] LustreError: 109753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3261.402383] Lustre: server umount lustre-MDT0001 complete [ 3271.944439] Lustre: server umount lustre-OST0000 complete [ 3282.388922] Lustre: server umount lustre-OST0001 complete [ 3290.900839] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 3305.648981] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 3345.905932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3346.139832] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3346.191610] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3346.269945] Lustre: lustre-MDT0000: new disk, initializing [ 3346.344966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3346.350083] Lustre: Skipped 3 previous similar messages [ 3346.373523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3351.566713] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3359.679606] Lustre: server umount lustre-MDT0000 complete [ 3371.678660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3372.199455] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3372.245382] Lustre: 116970:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3372.272470] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3377.593405] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3382.928367] Lustre: server umount lustre-MDT0000 complete [ 3393.148736] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 05:38:58 (1779701938) [ 3395.281783] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3397.263580] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 05:39:03 (1779701943) [ 3416.809496] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 3428.159853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3433.085801] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3442.449596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3442.603395] Lustre: 118989:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3442.738653] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3442.742718] Lustre: Skipped 1 previous similar message [ 3442.849793] Lustre: lustre-MDT0001: new disk, initializing [ 3442.967417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3442.980833] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3447.249806] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3451.778595] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3457.197297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3457.437221] Lustre: lustre-OST0000: new disk, initializing [ 3457.442487] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3457.447787] Lustre: 120530:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3458.630038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3458.645078] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3458.740484] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3463.014452] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3475.362815] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3475.707979] Lustre: lustre-OST0001: new disk, initializing [ 3475.721363] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3475.734534] Lustre: 121386:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3477.633770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3477.652588] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3477.740632] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3484.086669] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3495.696593] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3500.192419] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3907.624279] Lustre: 127792:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3907.633262] Lustre: 127792:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3907.640986] Lustre: 127792:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3907.655767] Lustre: 127792:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3907.663115] Lustre: 127792:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3907.669552] Lustre: 127792:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4186.962677] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:52:12 (1779702732) [ 4205.026213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4205.035152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4205.056878] Lustre: Skipped 8 previous similar messages [ 4205.071772] Lustre: Skipped 11 previous similar messages [ 4210.148414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4210.151192] Lustre: Skipped 1 previous similar message [ 4217.311423] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4217.537378] Lustre: server umount lustre-MDT0000 complete [ 4220.393649] LustreError: 122180:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4220.434899] LustreError: 122180:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 16 previous similar messages [ 4221.289409] LustreError: 121385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779702769 with bad export cookie 10218487268151831172 [ 4221.296058] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4221.302330] LustreError: 121385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4225.507991] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4225.541417] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4225.544132] Lustre: Skipped 4 previous similar messages [ 4228.081099] Lustre: server umount lustre-MDT0001 complete [ 4238.329193] Lustre: server umount lustre-OST0000 complete [ 4248.469136] Lustre: server umount lustre-OST0001 complete [ 4256.854496] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 4271.938701] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 4320.844353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4321.245328] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4321.289945] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4321.427211] Lustre: lustre-MDT0000: new disk, initializing [ 4321.606881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4321.615274] Lustre: Skipped 5 previous similar messages [ 4321.669765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4326.812111] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4339.949333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4340.156737] Lustre: 130837:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4340.251549] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4340.258054] Lustre: Skipped 1 previous similar message [ 4340.387630] Lustre: lustre-MDT0001: new disk, initializing [ 4340.559277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4340.572528] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4345.537328] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4354.262994] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4358.843691] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4366.376708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4366.679640] Lustre: lustre-OST0000: new disk, initializing [ 4366.681946] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4366.689774] Lustre: 132578:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4366.806580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4366.819439] Lustre: Skipped 1 previous similar message [ 4368.760439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4368.778331] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4368.892014] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4373.707170] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4383.870114] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4397.757899] Lustre: lustre-MDD0000: changelog on [ 4401.434658] Lustre: lustre-MDD0001: changelog on [ 6535.614445] Lustre: lustre-MDD0001: changelog off [ 6537.665726] Lustre: 130844:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6538.673377] Lustre: lustre-MDD0000: changelog off [ 6547.955564] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6547.956869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6547.973950] Lustre: Skipped 2 previous similar messages [ 6547.998804] Lustre: Skipped 3 previous similar messages [ 6552.927222] Lustre: server umount lustre-OST0000 complete [ 6558.192804] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6558.224690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6562.784660] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6562.807767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6562.843283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6564.729727] Lustre: server umount lustre-MDT0000 complete [ 6567.396902] LustreError: 131692:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6567.427773] LustreError: 131692:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6569.970567] LustreError: 130827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779705117 with bad export cookie 10218487268152610832 [ 6569.976437] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6569.985198] LustreError: 130827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6570.982557] Lustre: server umount lustre-MDT0001 complete [ 6590.243203] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 6593.999591] Key type lgssc unregistered [ 6594.390906] LNet: 141649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6594.399509] LNetError: 141649:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6594.413753] LNet: Removed LNI 192.168.202.105@tcp [ 6595.414431] Key type .llcrypt unregistered [ 6595.415785] Key type ._llcrypt unregistered [ 6621.478070] Key type ._llcrypt registered [ 6621.480118] Key type .llcrypt registered [ 6621.747482] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 6647.125721] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 6649.114026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6649.176142] alg: No test for adler32 (adler32-zlib) [ 6650.325403] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 6650.696518] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 6652.439264] Key type lgssc registered [ 6654.185790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6703.691654] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6703.760260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6705.363859] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6705.433601] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6705.556797] Lustre: lustre-MDT0000: new disk, initializing [ 6705.635455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6705.686878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6711.054244] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6723.436852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6723.658071] Lustre: 144827:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6723.762885] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6723.769537] Lustre: Skipped 1 previous similar message [ 6723.907662] Lustre: lustre-MDT0001: new disk, initializing [ 6724.146449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6724.237935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6724.251261] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6730.578190] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6739.841724] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6745.223717] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6754.383335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6754.770367] Lustre: lustre-OST0000: new disk, initializing [ 6754.776873] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6754.790629] Lustre: 146569:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6754.921788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6760.582257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6760.603681] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6760.824093] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6762.989962] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6772.612860] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6781.336876] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_module obdecho/obdecho [ 6783.788903] LustreError: 147444:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6783.800554] LustreError: 147444:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6785.940238] LustreError: 147540:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 6800.865536] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6800.883511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6800.927140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6801.891145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6801.911121] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6801.941657] Lustre: Skipped 1 previous similar message [ 6805.018529] Lustre: server umount lustre-MDT0000 complete [ 6809.635524] LustreError: 144817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779705357 with bad export cookie 17242862523184781110 [ 6809.656334] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6809.656560] LustreError: 144817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6810.033559] Lustre: server umount lustre-MDT0001 complete [ 6825.618928] Lustre: server umount lustre-OST0000 complete [ 6851.594946] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 6867.932080] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 6914.707666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6914.941544] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6914.965189] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6915.031757] Lustre: lustre-MDT0000: new disk, initializing [ 6915.138410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6915.156046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6920.114901] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6933.843900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6934.023824] Lustre: 151242:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6934.122129] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6934.128856] Lustre: Skipped 1 previous similar message [ 6934.245341] Lustre: lustre-MDT0001: new disk, initializing [ 6934.399902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6934.464359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6934.480183] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6940.272313] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6950.154809] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6955.675615] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6964.666047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6965.234665] Lustre: lustre-OST0000: new disk, initializing [ 6965.238518] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6965.248961] Lustre: 152988:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6965.390151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6972.019742] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6975.026658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6975.036157] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6975.227665] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6981.925902] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6988.271525] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6988.664596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6994.333933] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6994.713086] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7001.063610] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7001.069813] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7001.073839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7001.075888] Lustre: Skipped 1 previous similar message [ 7001.087220] Lustre: Skipped 3 previous similar messages [ 7003.463737] Lustre: server umount lustre-OST0000 complete [ 7011.312217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7011.313070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7011.335273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7011.345547] LustreError: Skipped 1 previous similar message [ 7011.369160] Lustre: Skipped 1 previous similar message [ 7014.362729] Lustre: server umount lustre-MDT0000 complete [ 7016.418430] LustreError: 151252:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7016.434593] LustreError: 151252:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7018.448318] LustreError: 152986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779705566 with bad export cookie 17242862523184782069 [ 7018.455655] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7018.784719] Lustre: server umount lustre-MDT0001 complete [ 7022.668758] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 06:39:28 (1779705568) [ 7024.960778] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7028.081482] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 06:39:33 (1779705573) [ 7037.685612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7038.117313] LustreError: 155256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7038.230297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7043.573893] LustreError: 155257:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7043.684136] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7048.683791] LustreError: 155256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7054.320074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7059.759803] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7067.015771] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7071.908064] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7080.034830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7080.287239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7080.290779] Lustre: Skipped 1 previous similar message [ 7087.798361] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7095.178586] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7107.342576] Lustre: server umount lustre-MDT0000 complete [ 7110.113432] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7110.127931] LustreError: 155257:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7110.138861] Lustre: Skipped 2 previous similar messages [ 7110.162314] LustreError: 155257:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7111.886821] LustreError: 155236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779705659 with bad export cookie 17242862523184782951 [ 7111.905963] LustreError: 155236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7111.926380] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7115.244575] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7118.522238] Lustre: server umount lustre-MDT0001 complete [ 7125.627375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7139.550581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7139.729706] LustreError: 158758:0:(llog.c:1646:llog_backup()) MGC192.168.202.105@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7139.735683] Lustre: 158758:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.202.105@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7141.857404] LustreError: 158763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7141.877565] LustreError: 158763:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7161.314192] LustreError: 158763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7161.330532] LustreError: 158763:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 7197.153187] LustreError: 158765:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7197.177528] LustreError: 158765:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 26 previous similar messages [ 7200.735688] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7200.741241] Lustre: 158758:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.105@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7241.711642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7246.521298] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7246.831868] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7254.915481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7262.711109] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7262.928694] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7269.883386] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7271.647782] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7275.325649] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7277.389812] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7298.531131] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7298.541810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7298.543761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7298.543769] Lustre: Skipped 2 previous similar messages [ 7298.550153] Lustre: Skipped 3 previous similar messages [ 7299.382489] Lustre: server umount lustre-MDT0000 complete [ 7303.401297] LustreError: 160266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779705851 with bad export cookie 17242862523184789405 [ 7303.404897] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7303.416675] LustreError: 160266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7303.662527] LustreError: 158765:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7303.698712] LustreError: 158765:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 41 previous similar messages [ 7303.927702] Lustre: server umount lustre-MDT0001 complete [ 7310.955449] Lustre: server umount lustre-OST0000 complete [ 7336.067467] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 7350.359860] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 7397.507485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7397.746356] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7397.775608] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7397.898399] Lustre: lustre-MDT0000: new disk, initializing [ 7398.044310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7398.055605] Lustre: Skipped 1 previous similar message [ 7398.078309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7403.548286] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7415.947458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7416.061601] Lustre: 163628:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7416.252367] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7419.649722] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7429.335991] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7435.362488] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7443.783935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7444.032556] Lustre: lustre-OST0000: new disk, initializing [ 7444.038970] Lustre: Skipped 1 previous similar message [ 7444.048533] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7444.051679] Lustre: Skipped 2 previous similar messages [ 7444.055203] Lustre: 165368:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7452.257158] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7453.808854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7453.822866] Lustre: Skipped 1 previous similar message [ 7453.834579] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7454.002381] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7462.044878] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7469.151571] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7469.528214] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7475.019581] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7475.397182] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7479.777282] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7479.783191] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7479.795225] LustreError: Skipped 1 previous similar message [ 7479.799109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7479.799117] Lustre: Skipped 2 previous similar messages [ 7479.852865] Lustre: Skipped 3 previous similar messages [ 7483.881326] Lustre: server umount lustre-OST0000 complete [ 7503.327235] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7503.926506] Lustre: server umount lustre-MDT0000 complete [ 7504.363253] LustreError: 163633:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7504.394361] LustreError: 163633:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7508.927527] LustreError: 164404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779706056 with bad export cookie 17242862523184806240 [ 7508.932294] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7508.943499] LustreError: 164404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7509.190902] Lustre: server umount lustre-MDT0001 complete [ 7521.181196] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 06:47:46 (1779706066) [ 7529.986639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7536.723501] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7548.024969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7554.238679] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7560.803655] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7566.038401] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7575.250056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7575.524183] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 7575.538444] Lustre: Skipped 7 previous similar messages [ 7583.265620] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7590.001365] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7604.418994] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 06:49:09 (1779706149) [ 7611.874417] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7611.896134] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7611.911238] Lustre: Skipped 2 previous similar messages [ 7616.139646] Lustre: server umount lustre-OST0000 complete [ 7626.988656] Lustre: server umount lustre-MDT0000 complete [ 7632.804614] LustreError: 167616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779706180 with bad export cookie 17242862523184807122 [ 7632.822942] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7633.518800] Lustre: server umount lustre-MDT0001 complete [ 7648.019451] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 7651.198556] Key type lgssc unregistered [ 7651.652670] LNet: 171009:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7651.680645] LNetError: 171009:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7651.705040] LNet: Removed LNI 192.168.202.105@tcp [ 7652.665428] Key type .llcrypt unregistered [ 7652.668174] Key type ._llcrypt unregistered [ 7676.538460] Key type ._llcrypt registered [ 7676.540918] Key type .llcrypt registered [ 7676.688399] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 7678.450865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7678.526835] alg: No test for adler32 (adler32-zlib) [ 7679.774994] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 7680.072831] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 7681.759175] Key type lgssc registered [ 7683.474598] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7693.710677] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7693.779228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7709.599581] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7714.719368] LustreError: 172238:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.105@tcp: failed processing log, type 4: rc = -110 [ 7719.839733] LustreError: 172238:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7719.851819] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7719.867104] LustreError: MGC192.168.202.105@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7719.881213] LustreError: 172238:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7719.897076] LustreError: 172238:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -22 [ 7719.909249] LustreError: 172238:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 7720.041441] Lustre: server umount lustre-OST0000 complete [ 7720.047734] LustreError: 172238:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7730.789695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7732.750703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7738.526735] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7742.945538] LustreError: 172739:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7742.998959] LustreError: 172739:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7748.072452] LustreError: 172738:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7749.261821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7749.941125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7755.313823] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7762.629859] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7767.238316] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7774.493393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7774.670147] LustreError: 174335:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7774.918597] LustreError: 174328:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7774.939300] Lustre: 174328:0:(mgc_request_server.c:748:mgc_llog_local_copy()) MGC192.168.202.105@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7775.218786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7782.415369] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7789.957208] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7796.195479] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7796.201139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7796.211474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7797.728350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7797.733170] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7797.778035] Lustre: Skipped 1 previous similar message [ 7799.650604] Lustre: server umount lustre-MDT0000 complete [ 7802.864211] LustreError: 173450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7802.905298] LustreError: 173450:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7804.227969] LustreError: 172720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779706352 with bad export cookie 3653895528731579080 [ 7804.253541] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7805.167356] Lustre: server umount lustre-MDT0001 complete [ 7819.473110] Lustre: server umount lustre-OST0000 complete [ 7830.721502] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 7842.708472] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 7896.809134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7897.287147] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7897.380647] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7897.500601] Lustre: lustre-MDT0000: new disk, initializing [ 7897.769326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7897.803111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7903.120793] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7917.085500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7917.278337] Lustre: 177722:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7917.422177] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7917.430888] Lustre: Skipped 1 previous similar message [ 7917.692896] Lustre: lustre-MDT0001: new disk, initializing [ 7917.818209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7917.862720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7917.877521] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7922.687893] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7932.744856] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7937.390241] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7946.420521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7946.907786] Lustre: lustre-OST0000: new disk, initializing [ 7946.920914] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7946.928924] Lustre: 179469:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7947.032594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7954.531673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7954.561245] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7954.703614] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7954.790718] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7963.479094] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7969.141519] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7969.618991] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7975.302717] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7975.610219] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7980.517692] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7980.521751] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7980.553734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7980.567399] Lustre: Skipped 3 previous similar messages [ 7985.639154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7985.649291] Lustre: Skipped 1 previous similar message [ 7990.755471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7990.772944] Lustre: Skipped 1 previous similar message [ 7992.287324] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7992.567711] Lustre: server umount lustre-OST0000 complete [ 7999.973465] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7999.985403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7999.998598] Lustre: Skipped 1 previous similar message [ 8000.008128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8003.840838] Lustre: server umount lustre-MDT0000 complete [ 8006.115281] LustreError: 177734:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8006.152523] LustreError: 177734:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8008.615990] LustreError: 177715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779706556 with bad export cookie 3653895528731579801 [ 8008.616303] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8008.635625] LustreError: 177715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8009.222236] Lustre: server umount lustre-MDT0001 complete [ 8019.580610] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 06:56:04 (1779706564) [ 8042.448987] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 8045.561621] Key type lgssc unregistered [ 8045.858802] LNet: 182255:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8045.869235] LNetError: 182255:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8045.887872] LNet: Removed LNI 192.168.202.105@tcp [ 8046.682938] Key type .llcrypt unregistered [ 8046.685830] Key type ._llcrypt unregistered [ 8069.575463] Key type ._llcrypt registered [ 8069.578413] Key type .llcrypt registered [ 8069.760076] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 8071.045344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8071.192406] alg: No test for adler32 (adler32-zlib) [ 8072.454961] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 8072.682650] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 8074.367203] Key type lgssc registered [ 8075.954558] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8085.153062] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8085.210671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8086.994636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8092.447141] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8096.246200] LustreError: 183460:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8096.276776] LustreError: 183460:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8101.356384] LustreError: 183459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8104.068673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8104.570525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8110.159413] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8118.055067] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8123.193863] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8132.846970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8133.108125] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8133.115653] LustreError: 185098:0:(llog.c:1687:llog_backup()) MGC192.168.202.105@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8133.130821] Lustre: 185098:0:(mgc_request_server.c:745:mgc_llog_local_copy()) MGC192.168.202.105@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8133.313789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8140.506657] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8148.646719] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8155.106480] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8155.110248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8155.144657] Lustre: Skipped 1 previous similar message [ 8157.469489] Lustre: server umount lustre-MDT0000 complete [ 8160.230720] LustreError: 184170:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8160.274607] LustreError: 184170:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8161.805797] LustreError: 183441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779706709 with bad export cookie 8806061727356021348 [ 8161.846259] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8162.214803] Lustre: server umount lustre-MDT0001 complete [ 8176.615835] Lustre: server umount lustre-OST0000 complete [ 8189.003685] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 8202.672767] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 8250.125803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8250.572436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8250.659188] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8250.776668] Lustre: lustre-MDT0000: new disk, initializing [ 8250.957571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8251.055664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8256.186912] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8269.039940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8269.167224] Lustre: 188477:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8269.215735] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8269.218603] Lustre: Skipped 1 previous similar message [ 8269.221616] LustreError: 189243:0:(llog.c:1687:llog_backup()) MGC192.168.202.105@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8269.228883] LustreError: 189243:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 8269.234606] Lustre: 189243:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.202.105@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8269.254479] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8269.259381] Lustre: Skipped 1 previous similar message [ 8269.351705] Lustre: lustre-MDT0001: new disk, initializing [ 8269.480946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8269.552283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8269.594934] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8274.515563] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8284.740031] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8289.766058] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8298.274733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8298.669364] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8298.675992] Lustre: Skipped 1 previous similar message [ 8298.682903] LustreError: 190195:0:(llog.c:1687:llog_backup()) MGC192.168.202.105@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8298.700210] LustreError: 190195:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 8298.707950] Lustre: 190195:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.202.105@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8298.730970] Lustre: 190195:0:(mgc_request_server.c:768:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8298.768954] Lustre: lustre-OST0000: new disk, initializing [ 8298.777852] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8298.795209] Lustre: 190220:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8298.935591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8305.708991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8305.728229] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8305.869393] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8306.087503] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8316.663492] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8322.074742] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8322.433121] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8328.532140] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8328.844765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8331.762841] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8331.774779] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8331.783255] Lustre: Skipped 1 previous similar message [ 8331.788139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8331.793164] Lustre: Skipped 2 previous similar messages [ 8336.864289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8336.876029] Lustre: Skipped 1 previous similar message [ 8337.935020] Lustre: server umount lustre-OST0000 complete [ 8345.057140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8345.073931] Lustre: Skipped 1 previous similar message [ 8345.092043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8349.709553] Lustre: server umount lustre-MDT0000 complete [ 8352.233411] LustreError: 188490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8352.270430] LustreError: 188490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8354.701701] LustreError: 188470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779706902 with bad export cookie 8806061727356022062 [ 8354.711283] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8354.721446] LustreError: 188470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8355.566581] Lustre: server umount lustre-MDT0001 complete [ 8366.608976] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 07:01:52 (1779706912) [ 8386.430437] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 8398.911563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8399.616461] LustreError: 193197:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8399.705232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8405.173535] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8410.081316] LustreError: 193198:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8410.101744] LustreError: 193198:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8416.566851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8422.718622] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8425.831819] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8433.390201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8433.866798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8433.870841] Lustre: Skipped 1 previous similar message [ 8440.578909] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8450.895639] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8451.094629] Lustre: lustre-OST0001: new disk, initializing [ 8451.098659] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8451.104147] Lustre: 195453:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8457.958921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8457.982860] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8458.227995] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8458.513959] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8470.838465] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8480.960778] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8483.694543] Lustre: DEBUG MARKER: ADD OST3 [ 8490.135069] loop: module loaded [ 8490.263887] loop0: detected capacity change from 0 to 204800000 [ 8490.576920] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8490.595066] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8490.608776] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8498.663777] Lustre: DEBUG MARKER: STOP OST3 [ 8499.198505] loop0: detected capacity change from 0 to 204800000 [ 8506.550901] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8506.845267] Lustre: lustre-OST0003: new disk, initializing [ 8506.857360] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8506.864697] Lustre: 197508:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8507.072993] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8507.082354] Lustre: Skipped 1 previous similar message [ 8508.354192] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8508.570511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8508.584890] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8508.660177] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8509.788789] Lustre: Failing over lustre-OST0003 [ 8510.262914] Lustre: server umount lustre-OST0003 complete [ 8510.382830] LustreError: 197600:0:(osp_precreate.c:1281:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8513.514542] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8513.530048] Lustre: Skipped 1 previous similar message [ 8513.537167] LustreError: 194651:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8513.551066] LustreError: 194651:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8514.711699] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8525.096506] Lustre: DEBUG MARKER: START OST3 again [ 8535.053941] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8535.508599] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8537.067408] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8537.609165] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8537.612329] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8542.824557] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8566.983688] Lustre: Failing over lustre-OST0003 [ 8567.151130] Lustre: server umount lustre-OST0003 complete [ 8568.301082] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8568.315311] LustreError: Skipped 1 previous similar message [ 8568.317364] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8568.332967] Lustre: Skipped 1 previous similar message [ 8568.345886] LustreError: 194651:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8568.388331] LustreError: 194651:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 8586.722981] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8586.735599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8586.739484] Lustre: Skipped 3 previous similar messages [ 8586.757181] Lustre: Skipped 5 previous similar messages [ 8591.345556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8591.355406] LustreError: Skipped 1 previous similar message [ 8592.986504] Lustre: server umount lustre-MDT0000 complete [ 8597.161943] LustreError: 194297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779707145 with bad export cookie 8806061727356022944 [ 8597.170064] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8597.175913] LustreError: 194297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8602.095221] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8602.105531] Lustre: Skipped 5 previous similar messages [ 8603.661369] Lustre: server umount lustre-MDT0001 complete [ 8609.291657] Lustre: server umount lustre-OST0000 complete [ 8613.717627] Lustre: server umount lustre-OST0001 complete [ 8620.870282] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 8630.558714] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 8677.422652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8677.799307] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8677.828527] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8677.964616] Lustre: lustre-MDT0000: new disk, initializing [ 8678.195043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8678.208118] Lustre: Skipped 1 previous similar message [ 8678.262435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8684.180595] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8696.144712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8696.344511] Lustre: 202838:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8696.440756] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8696.450501] Lustre: Skipped 1 previous similar message [ 8696.601351] Lustre: lustre-MDT0001: new disk, initializing [ 8696.842752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8696.857670] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8701.581877] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8711.100372] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8716.318279] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8725.054233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8725.492906] Lustre: 204580:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8727.291504] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8727.369572] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8733.857528] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8743.261680] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8748.810657] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8749.134760] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8754.873719] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8755.216917] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8758.248152] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8758.269848] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8758.281692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8758.286714] Lustre: Skipped 4 previous similar messages [ 8758.301379] Lustre: Skipped 1 previous similar message [ 8763.437746] LustreError: 204668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8763.474207] LustreError: 204668:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 16 previous similar messages [ 8763.636751] Lustre: server umount lustre-OST0000 complete [ 8773.600742] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8773.610758] LustreError: Skipped 1 previous similar message [ 8775.749316] Lustre: server umount lustre-MDT0000 complete [ 8780.424885] LustreError: 202830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779707328 with bad export cookie 8806061727356025240 [ 8780.443679] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8780.802685] Lustre: server umount lustre-MDT0001 complete [ 8785.701274] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 07:08:51 (1779707331) [ 8806.811899] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 8822.956313] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 8872.319975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8872.875878] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8872.943594] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8872.959612] Lustre: Skipped 1 previous similar message [ 8873.117256] Lustre: lustre-MDT0000: new disk, initializing [ 8873.124490] Lustre: Skipped 1 previous similar message [ 8873.374553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8873.389335] Lustre: Skipped 1 previous similar message [ 8879.210415] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8892.844528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8893.001183] Lustre: 209031:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8893.267075] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8899.979093] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8909.650851] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8914.933853] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8923.159798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8923.434333] Lustre: 210776:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8925.131802] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8930.729237] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8939.035399] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8944.392924] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8944.880945] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8950.496026] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8950.822245] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8955.876934] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8955.879804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8955.888282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8955.888290] Lustre: Skipped 3 previous similar messages [ 8955.891778] LustreError: Skipped 1 previous similar message [ 8955.950101] Lustre: Skipped 3 previous similar messages [ 8959.652724] Lustre: server umount lustre-OST0000 complete [ 8978.399135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8978.599931] Lustre: server umount lustre-MDT0000 complete [ 8980.449464] LustreError: 209039:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8980.465978] LustreError: 209039:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8983.685494] LustreError: 209024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779707531 with bad export cookie 8806061727356026115 [ 8983.687906] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8983.709476] LustreError: 209024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8984.161814] Lustre: server umount lustre-MDT0001 complete [ 8994.977163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8995.894827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8995.917742] Lustre: Skipped 5 previous similar messages [ 9001.317531] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 9011.343402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9017.264534] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 9024.254180] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9028.169820] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9037.815882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9045.261432] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 9051.080498] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9061.858131] LustreError: lustre-OST0000-osc-MDT0001: operation ost_create to node 0@lo failed: rc = -107 [ 9061.865137] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9061.870485] LustreError: Skipped 2 previous similar messages [ 9061.889590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9061.901693] Lustre: Skipped 3 previous similar messages [ 9061.907398] Lustre: Skipped 8 previous similar messages [ 9076.192203] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9076.638189] Lustre: server umount lustre-OST0000 complete [ 9090.531815] Lustre: server umount lustre-MDT0000 complete [ 9095.286125] LustreError: 212827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779707643 with bad export cookie 8806061727356027004 [ 9095.290366] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9095.295895] LustreError: 212827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9095.307185] LustreError: 213559:0:(osp_precreate.c:652:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 9095.314734] LustreError: 213559:0:(osp_precreate.c:1356:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 9095.981747] Lustre: server umount lustre-MDT0001 complete [ 9191.046272] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 9194.288245] Key type lgssc unregistered [ 9194.628388] LNet: 216153:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9194.653861] LNetError: 216153:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9194.670034] LNet: Removed LNI 192.168.202.105@tcp [ 9195.749815] Key type .llcrypt unregistered [ 9195.756883] Key type ._llcrypt unregistered [ 9227.658753] Key type ._llcrypt registered [ 9227.660950] Key type .llcrypt registered [ 9227.774651] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 9246.864345] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 9248.452712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9248.490653] alg: No test for adler32 (adler32-zlib) [ 9249.707504] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 9250.054434] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 9251.807289] Key type lgssc registered [ 9253.591727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9295.517109] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9295.538433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9296.867237] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9296.898735] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9296.988598] Lustre: lustre-MDT0000: new disk, initializing [ 9297.112043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9297.133675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9302.755360] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 9315.565689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9315.755902] Lustre: 219481:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9315.880315] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9315.903250] Lustre: Skipped 1 previous similar message [ 9315.992815] Lustre: lustre-MDT0001: new disk, initializing [ 9316.124306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9316.172274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9316.186706] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9321.671248] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 9330.182125] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9334.599741] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9342.466228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9342.920927] Lustre: lustre-OST0000: new disk, initializing [ 9342.932762] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9342.942757] Lustre: 221222:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9343.093288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9349.699369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9349.718904] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9349.879279] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9352.346794] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 9361.608986] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9367.531407] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9367.737317] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9373.029100] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9373.307340] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9380.836158] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9380.840696] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9380.852996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9380.858184] Lustre: Skipped 1 previous similar message [ 9385.961376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9385.973821] Lustre: Skipped 2 previous similar messages [ 9390.047274] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9390.416320] Lustre: server umount lustre-OST0000 complete [ 9396.195193] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9396.201170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9396.221299] Lustre: Skipped 1 previous similar message [ 9396.241111] Lustre: Skipped 1 previous similar message [ 9401.312911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9401.323682] Lustre: Skipped 1 previous similar message [ 9402.406433] Lustre: server umount lustre-MDT0000 complete [ 9406.434484] LustreError: 220268:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9406.446202] LustreError: 220268:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9406.794725] LustreError: 219472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779707954 with bad export cookie 5039313120495312895 [ 9406.816467] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9407.387797] Lustre: server umount lustre-MDT0001 complete [ 9431.374582] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 9443.409720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9443.938227] LustreError: 224013:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9444.027828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9448.709739] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 9449.441295] LustreError: 224014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9449.479558] LustreError: 224014:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9453.545053] LustreError: 224013:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9458.660906] LustreError: 224014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9459.413256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9459.760714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9465.249173] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 9469.695887] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9477.720397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9478.141937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9486.288252] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 9496.826898] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9497.008239] Lustre: lustre-OST0001: new disk, initializing [ 9497.012783] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9497.018292] Lustre: 226270:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9497.111429] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9503.937181] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 9504.416821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9504.437819] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9504.650356] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9517.656347] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9526.717887] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9541.360513] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 07:21:26 (1779708086) [ 9545.717050] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9545.729073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9545.755865] Lustre: Skipped 3 previous similar messages [ 9545.774790] Lustre: Skipped 3 previous similar messages [ 9550.564721] Lustre: server umount lustre-MDT0000 complete [ 9550.826325] LustreError: 224013:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9550.859894] LustreError: 224013:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9554.848713] LustreError: 223996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779708102 with bad export cookie 5039313120495313777 [ 9554.856060] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9554.862379] LustreError: 223996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9555.594826] Lustre: server umount lustre-MDT0001 complete [ 9570.876350] Lustre: server umount lustre-OST0000 complete [ 9571.106611] Lustre: 217699:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779708103/real 1779708103] req@ffff8ea7d0355180 x1866158883067904/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779708119 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9571.156315] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9575.238105] Lustre: server umount lustre-OST0001 complete [ 9584.870811] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 9600.102270] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 9645.214210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9645.521982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9645.558960] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9645.636725] Lustre: lustre-MDT0000: new disk, initializing [ 9645.782363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9645.803327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9651.645606] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 9664.336091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9664.488332] Lustre: 230192:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9664.550060] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9664.554372] Lustre: Skipped 1 previous similar message [ 9664.627214] Lustre: lustre-MDT0001: new disk, initializing [ 9664.766925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9664.782403] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9669.463939] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 9678.642066] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9684.157442] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9693.979228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9694.425851] Lustre: lustre-OST0000: new disk, initializing [ 9694.428778] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9694.433240] Lustre: 231936:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9695.865861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9695.875465] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9696.081645] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9702.166595] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 9711.604527] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9717.156851] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9717.466684] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9722.814569] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9723.030604] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9726.948898] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9726.955078] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9726.974243] LustreError: Skipped 1 previous similar message [ 9726.976787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9727.004130] Lustre: Skipped 1 previous similar message [ 9727.020602] Lustre: Skipped 1 previous similar message [ 9731.500792] Lustre: server umount lustre-OST0000 complete [ 9737.201762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9743.504675] Lustre: server umount lustre-MDT0000 complete [ 9746.476442] LustreError: 230204:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9746.535495] LustreError: 230204:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9748.631438] LustreError: 230185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779708296 with bad export cookie 5039313120495315912 [ 9748.665146] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9749.512305] Lustre: server umount lustre-MDT0001 complete [ 9761.453464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9762.371046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9762.373660] Lustre: Skipped 2 previous similar messages [ 9766.746612] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 9776.006356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9780.658917] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 9788.922110] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9793.670923] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9802.070737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9809.913435] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 9816.716126] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9965.543850] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9965.581024] Lustre: Skipped 1 previous similar message [ 9965.584141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9965.594348] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9965.619963] Lustre: Skipped 5 previous similar messages [ 9971.079853] Lustre: server umount lustre-OST0000 complete [ 9980.910471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9982.630955] Lustre: server umount lustre-MDT0000 complete [ 9986.017990] LustreError: 234718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9986.048345] LustreError: 234718:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9986.866361] LustreError: 233988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779708534 with bad export cookie 5039313120495316794 [ 9986.867963] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9986.902090] LustreError: 233988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9987.418797] Lustre: server umount lustre-MDT0001 complete [10084.534745] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [10087.745465] Key type lgssc unregistered [10088.121792] LNet: 237255:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10088.149660] LNetError: 237255:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10088.178962] LNet: Removed LNI 192.168.202.105@tcp [10089.178205] Key type .llcrypt unregistered [10089.183394] Key type ._llcrypt unregistered [10117.527710] Key type ._llcrypt registered [10117.536535] Key type .llcrypt registered [10117.692558] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [10135.367056] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [10136.490813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10136.645639] alg: No test for adler32 (adler32-zlib) [10137.796220] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [10138.012912] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [10139.727217] Key type lgssc registered [10141.266693] Lustre: Echo OBD driver; http://www.lustre.org/ [10191.466746] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [10203.404120] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10203.443308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10204.851163] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10204.877861] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10205.001725] Lustre: lustre-MDT0000: new disk, initializing [10205.078691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10205.093981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10210.355321] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [10221.461278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10221.723430] Lustre: 241278:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10221.789301] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10221.794244] Lustre: Skipped 1 previous similar message [10221.882959] Lustre: lustre-MDT0001: new disk, initializing [10221.990678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10222.040574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10222.054716] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10226.515281] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [10231.940199] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10238.737183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10239.062578] Lustre: lustre-OST0000: new disk, initializing [10239.066539] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10239.076247] Lustre: 242877:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10239.162696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10245.224024] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10245.243345] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10245.367702] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10245.779674] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [10259.383817] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10259.570145] Lustre: lustre-OST0001: new disk, initializing [10259.574742] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10259.583927] Lustre: 243730:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10259.777262] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10266.929455] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [10267.770852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10267.786909] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10267.876359] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10278.708358] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10283.353519] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10298.855374] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10298.883498] Lustre: Skipped 1 previous similar message [10298.888427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10300.432127] Lustre: server umount lustre-MDT0000 complete [10305.555392] LustreError: 242065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779708853 with bad export cookie 8525610772416521169 [10305.559267] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10305.571517] LustreError: 242065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10306.318682] Lustre: server umount lustre-MDT0001 complete [10321.549869] Lustre: server umount lustre-OST0000 complete [10324.256133] Lustre: 238801:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779708856/real 1779708857] req@ffff8ea6c30ff100 x1866159814161280/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779708872 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10324.301968] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10324.337464] Lustre: Skipped 2 previous similar messages [10326.497527] Lustre: 238803:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779708858/real 1779708858] req@ffff8ea801d38000 x1866159814161536/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779708874 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10327.116451] Lustre: server umount lustre-OST0001 complete [10344.838616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10356.086529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10366.075914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10376.038517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10400.037916] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [10413.451635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10413.473456] Lustre: lustre-MDT0000: reset Object Index mappings [10413.976382] LustreError: 248709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10413.996778] LustreError: 248709:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10414.105280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10419.068193] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [10419.178819] LustreError: 248710:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10419.206264] LustreError: 248710:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10423.266626] LustreError: 248709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10428.388367] LustreError: 248710:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10429.725476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10429.763725] Lustre: lustre-MDT0001: reset Object Index mappings [10430.302092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10434.898077] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [10438.371777] Lustre: 249817:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10445.390230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10452.814430] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [10455.978184] LustreError: 250170:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10459.095985] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10462.978547] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10463.457596] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10463.463233] Lustre: Skipped 1 previous similar message [10468.902530] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10471.086462] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [10481.042957] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10485.677271] Lustre: 251655:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10491.872544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10491.876978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10491.892383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10491.894807] Lustre: Skipped 3 previous similar messages [10494.438058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10494.439794] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10494.455160] Lustre: Skipped 1 previous similar message [10494.465760] Lustre: Skipped 1 previous similar message [10496.063588] Lustre: server umount lustre-MDT0000 complete [10499.567924] LustreError: 248705:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10499.599982] LustreError: 248705:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10500.764492] LustreError: 248691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779709048 with bad export cookie 8525610772416527959 [10500.796264] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10500.799961] LustreError: 248691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10501.369124] Lustre: server umount lustre-MDT0001 complete [10516.666519] Lustre: server umount lustre-OST0000 complete [10531.420430] Lustre: server umount lustre-OST0001 complete [10556.634350] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 07:38:21 (1779709101) [10575.897334] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [10586.961318] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [10628.774605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10629.186928] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10629.221936] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10629.439100] Lustre: lustre-MDT0000: new disk, initializing [10629.628176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10629.665553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10634.718349] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [10644.790737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10644.958117] Lustre: 255880:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10645.002494] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10645.005885] Lustre: Skipped 1 previous similar message [10645.068532] Lustre: lustre-MDT0001: new disk, initializing [10645.186216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10645.207028] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10649.633068] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [10657.538442] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10662.216249] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10669.595872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10669.925585] Lustre: lustre-OST0000: new disk, initializing [10669.929249] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10669.936937] Lustre: 257621:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10671.117956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10671.127695] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10671.291124] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10676.211678] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [10686.188795] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10692.574468] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10692.909287] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10697.766855] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10697.959354] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10701.797187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10701.810271] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10701.831908] Lustre: Skipped 1 previous similar message [10701.838989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10706.067187] Lustre: server umount lustre-OST0000 complete [10712.033119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10712.046319] LustreError: Skipped 1 previous similar message [10712.050771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10712.071711] Lustre: Skipped 1 previous similar message [10712.080823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10712.083121] Lustre: Skipped 2 previous similar messages [10716.192685] Lustre: server umount lustre-MDT0000 complete [10717.667324] LustreError: 256758:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10717.701648] LustreError: 256758:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10720.285852] LustreError: 255872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779709268 with bad export cookie 8525610772416530052 [10720.299534] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10720.301611] LustreError: 255872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10720.765137] Lustre: server umount lustre-MDT0001 complete [10743.024720] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [10755.185736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10755.805133] LustreError: 260391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10755.865578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10755.871443] Lustre: Skipped 2 previous similar messages [10759.684728] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [10768.204285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10773.377711] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [10776.554285] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10783.913968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10790.314810] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [10799.000861] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10799.228819] Lustre: lustre-OST0001: new disk, initializing [10799.235032] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10799.242522] Lustre: 262650:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10800.853321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10800.865160] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10800.977924] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10805.539440] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [10815.743649] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10819.739167] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10825.174180] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10825.194382] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10831.842081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10831.849507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10831.855648] Lustre: Skipped 4 previous similar messages [10831.867291] Lustre: Skipped 4 previous similar messages [10836.914293] Lustre: server umount lustre-MDT0000 complete [10836.967844] LustreError: 260392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10836.984609] LustreError: 260392:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10841.109066] LustreError: 261867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779709389 with bad export cookie 8525610772416530934 [10841.124393] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10841.136451] LustreError: 261867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10841.531346] Lustre: server umount lustre-MDT0001 complete [10856.769318] Lustre: server umount lustre-OST0000 complete [10857.439137] Lustre: 238802:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779709389/real 1779709389] req@ffff8ea6cbeb3800 x1866159814314112/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779709405 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10861.018229] Lustre: server umount lustre-OST0001 complete [10879.108286] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [10889.786399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10890.276322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10890.282466] Lustre: Skipped 3 previous similar messages [10895.175093] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [10904.204738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10909.190889] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [10912.239884] Lustre: 266772:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10918.837147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10925.735605] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [10935.102393] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10936.455145] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10936.479173] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10942.412527] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [10950.653464] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10954.879202] Lustre: 268615:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10960.286978] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10960.415352] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10971.072933] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 07:45:16 (1779709516) [10988.008206] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10988.017809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10988.041041] Lustre: Skipped 4 previous similar messages [10988.082594] Lustre: Skipped 3 previous similar messages [10993.687533] Lustre: server umount lustre-MDT0000 complete [10998.046374] LustreError: 268617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779709545 with bad export cookie 8525610772416537269 [10998.074386] LustreError: 268617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10998.080311] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10998.273993] LustreError: 265661:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10998.305949] LustreError: 265661:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [11004.540297] Lustre: server umount lustre-MDT0001 complete [11023.327252] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11023.408585] Lustre: server umount lustre-OST0000 complete [11027.381720] Lustre: server umount lustre-OST0001 complete [11034.476850] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [11045.289252] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [11095.219499] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [11106.716642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11107.006728] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11107.043867] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11107.111319] Lustre: lustre-MDT0000: new disk, initializing [11107.213110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11111.409640] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [11121.703890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11121.818828] Lustre: 272809:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11122.148943] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11126.471433] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [11138.607538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11138.956402] Lustre: 274410:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11141.191965] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11146.109612] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [11158.239978] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11158.462125] Lustre: 275263:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11158.641543] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11158.650025] Lustre: Skipped 6 previous similar messages [11160.095219] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11160.105814] Lustre: Skipped 1 previous similar message [11160.208857] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11165.924406] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [11175.505835] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11179.847228] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11179.851994] Lustre: Skipped 1 previous similar message [11199.818118] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 07:49:04 (1779709744) [11204.069896] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11204.083400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11204.099136] Lustre: Skipped 2 previous similar messages [11204.122109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11204.134703] Lustre: Skipped 8 previous similar messages [11208.930745] Lustre: server umount lustre-MDT0000 complete [11213.007459] LustreError: 273588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779709760 with bad export cookie 8525610772416545144 [11213.014431] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11213.021801] LustreError: 273588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11213.443688] Lustre: server umount lustre-MDT0001 complete [11227.868441] Lustre: server umount lustre-OST0000 complete [11242.002531] Lustre: server umount lustre-OST0001 complete [11251.930083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11257.659747] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [11257.827712] LustreError: 277494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11257.860759] LustreError: 277494:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [11267.380688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11272.721525] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [11279.595412] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11284.762868] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11293.914509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11302.704106] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [11308.920484] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11330.498176] loop1: detected capacity change from 0 to 204800000 [11330.813802] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11330.826975] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11330.839688] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11336.215744] loop1: detected capacity change from 0 to 204800000 [11342.344996] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11342.454603] LustreError: 277481:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs14139: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11342.464487] LustreError: 277481:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs14139 [11342.468968] LustreError: 277481:0:(mgs_handler.c:517:mgs_target_reg()) fs14139-MDT0001: Error checking target index: rc = -13 [11342.483176] LustreError: fs14139-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11342.493868] LustreError: 280923:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [11342.499455] LustreError: 280923:0:(tgt_mount.c:2084:server_put_super()) no obd fs14139-MDT0001 [11342.506919] LustreError: 280923:0:(tgt_mount.c:116:server_deregister_mount()) fs14139-MDT0001 not registered [11342.534379] Lustre: server umount fs14139-MDT0001 complete [11342.537474] LustreError: 280923:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [11352.841995] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11352.947521] Lustre: Setting parameter fs14139-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs14139-MDT0001 [11353.061936] Lustre: srv-fs14139-MDT0001: No data found on store. Initialize space. [11353.086535] Lustre: Skipped 4 previous similar messages [11353.239069] Lustre: fs14139-MDT0001: new disk, initializing [11353.243730] Lustre: Skipped 3 previous similar messages [11353.390836] Lustre: srv-fs14139-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11353.401490] Lustre: 281390:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs14139-MDT0001-mdtlov: can't create new llog: rc = -115 [11353.414121] LustreError: 281390:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs14139-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11360.065570] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [11367.646664] Lustre: server umount fs14139-MDT0001 complete [11379.697739] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11382.076472] loop1: detected capacity change from 0 to 204800000 [11388.695947] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11389.054498] Lustre: 282940:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs14139-MDT0001-mdtlov: can't create new llog: rc = -115 [11389.061719] LustreError: 282940:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs14139-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11394.426949] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [11395.053482] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11399.068546] Lustre: server umount fs14139-MDT0001 complete [11410.403035] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11410.417075] Lustre: Skipped 3 previous similar messages [11410.418200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11410.431497] Lustre: Skipped 6 previous similar messages [11414.780825] Lustre: server umount lustre-MDT0000 complete [11424.096669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11424.160389] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11429.379190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11429.506497] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [11437.602705] loop1: detected capacity change from 0 to 204800000 [11443.638380] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11443.980449] Lustre: 285354:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs14139-MDT0001-mdtlov: can't create new llog: rc = -115 [11444.001365] LustreError: 285354:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs14139-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11448.673652] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [11453.557273] Lustre: server umount fs14139-MDT0001 complete [11480.565207] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11485.617792] Lustre: server umount lustre-OST0000 complete [11496.299848] Lustre: server umount lustre-MDT0000 complete [11500.518398] LustreError: 277473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779710048 with bad export cookie 8525610772416547825 [11500.526982] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11500.530032] LustreError: 277473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11501.075266] Lustre: server umount lustre-MDT0001 complete [11514.200260] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [11517.165069] Key type lgssc unregistered [11517.463288] LNet: 287666:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11517.479465] LNetError: 287666:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11517.502740] LNet: Removed LNI 192.168.202.105@tcp [11518.432317] Key type .llcrypt unregistered [11518.434127] Key type ._llcrypt unregistered [11540.756924] Key type ._llcrypt registered [11540.759170] Key type .llcrypt registered [11540.865588] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [11542.475651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11542.743263] alg: No test for adler32 (adler32-zlib) [11543.809905] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [11544.069852] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [11545.791335] Key type lgssc registered [11546.729530] Lustre: Echo OBD driver; http://www.lustre.org/ [11555.305987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11555.355806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11557.293671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11562.177289] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [11567.584949] LustreError: 288921:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11567.620327] LustreError: 288921:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11572.350703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11572.710268] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11572.940184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11577.919386] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [11583.450351] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11587.730900] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11597.727849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11597.820328] LustreError: 290528:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11597.856538] LustreError: 290528:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11598.146687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11602.229933] LustreError: 290525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11602.253973] LustreError: 290525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11604.929799] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [11607.543562] LustreError: 290526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11607.577906] LustreError: 290526:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11611.488735] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11612.658811] LustreError: 290526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11612.679626] LustreError: 290526:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11622.896109] LustreError: 290528:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11622.917291] LustreError: 290528:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11632.306589] loop1: detected capacity change from 0 to 204800000 [11632.520432] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11632.535358] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11632.553427] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11638.419700] loop1: detected capacity change from 0 to 204800000 [11640.121524] LustreError: 290528:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11640.164584] LustreError: 290528:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [11644.749764] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11644.981811] Lustre: fs9417-OST0000: new disk, initializing [11644.988692] Lustre: srv-fs9417-OST0000: No data found on store. Initialize space. [11644.994392] Lustre: 292365:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11645.072898] Lustre: fs9417-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11652.040606] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [11661.401967] Lustre: server umount fs9417-OST0000 complete [11674.083170] LustreError: 290526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11674.110660] LustreError: 290526:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [11675.578821] loop1: detected capacity change from 0 to 204800000 [11679.247978] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11682.712431] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11683.322033] Lustre: fs9417-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11684.336336] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11690.340241] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [11700.330326] loop2: detected capacity change from 0 to 204800000 [11700.599213] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11700.618343] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11700.636177] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11707.579700] loop2: detected capacity change from 0 to 204800000 [11714.916552] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11715.015468] LustreError: fs9417-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11715.026049] LustreError: fs9417-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11715.034710] LustreError: 295373:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [11715.042996] LustreError: 295373:0:(tgt_mount.c:2084:server_put_super()) no obd fs9417-OST0001 [11715.055229] LustreError: 295373:0:(tgt_mount.c:116:server_deregister_mount()) fs9417-OST0001 not registered [11715.079783] Lustre: server umount fs9417-OST0001 complete [11715.082533] LustreError: 295373:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [11726.400978] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11726.502362] Lustre: fs9417-OST0001: new disk, initializing [11726.505657] Lustre: srv-fs9417-OST0001: No data found on store. Initialize space. [11726.512219] Lustre: 295925:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11726.618827] Lustre: fs9417-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11732.870348] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [11740.641950] LustreError: 290525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11740.689432] LustreError: 290525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [11741.724399] Lustre: server umount fs9417-OST0000 complete [11751.345968] Lustre: server umount fs9417-OST0001 complete [11764.727609] loop1: detected capacity change from 0 to 204800000 [11770.948865] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11771.265344] Lustre: fs9417-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11777.604871] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [11785.013732] loop2: detected capacity change from 0 to 204800000 [11791.433197] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11797.943029] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [11807.199576] Lustre: server umount fs9417-OST0000 complete [11816.888764] Lustre: server umount fs9417-OST0001 complete [11832.813079] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11832.815818] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11832.822234] LustreError: Skipped 2 previous similar messages [11832.827294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11832.850015] Lustre: Skipped 1 previous similar message [11837.935205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11837.939397] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11837.949552] Lustre: Skipped 2 previous similar messages [11838.541161] Lustre: server umount lustre-OST0000 complete [11848.161167] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11848.162620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11848.180168] Lustre: Skipped 1 previous similar message [11848.193528] Lustre: Skipped 1 previous similar message [11851.009102] Lustre: server umount lustre-MDT0000 complete [11855.195112] LustreError: 288901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779710403 with bad export cookie 5364307313006523372 [11855.209093] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11855.511928] Lustre: server umount lustre-MDT0001 complete [11869.351274] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [11873.025188] Key type lgssc unregistered [11873.295723] LNet: 301785:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11873.303598] LNetError: 301785:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11873.334181] LNet: Removed LNI 192.168.202.105@tcp [11874.220213] Key type .llcrypt unregistered [11874.222062] Key type ._llcrypt unregistered [11909.846279] Key type ._llcrypt registered [11909.847814] Key type .llcrypt registered [11910.032207] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [11926.875830] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [11928.192192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11928.377732] alg: No test for adler32 (adler32-zlib) [11929.465973] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [11929.885038] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [11931.647674] Key type lgssc registered [11932.890357] Lustre: Echo OBD driver; http://www.lustre.org/ [11970.599437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11970.646932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11972.062067] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11972.112566] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11972.241736] Lustre: lustre-MDT0000: new disk, initializing [11972.353193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11972.376741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11978.333647] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [11989.818011] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11989.966743] Lustre: 305631:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11990.048268] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11990.052856] Lustre: Skipped 1 previous similar message [11990.184891] Lustre: lustre-MDT0001: new disk, initializing [11990.311593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11990.341991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11990.351654] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11994.804516] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [12002.959345] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12007.634718] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12015.281259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12015.599982] Lustre: lustre-OST0000: new disk, initializing [12015.602352] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12015.609297] Lustre: 307372:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12015.668723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12023.108607] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [12023.385188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12023.406967] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12023.603582] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12032.876980] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12035.077425] Lustre: Mounted lustre-client [12035.998849] LustreError: 308126:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff8ea7c86d3800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [12036.078682] Lustre: Unmounted lustre-client [12049.384724] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12049.385984] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12049.395182] LustreError: Skipped 1 previous similar message [12049.401695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12049.425558] Lustre: Skipped 1 previous similar message [12049.450653] Lustre: Skipped 1 previous similar message [12050.433257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12050.443337] Lustre: Skipped 1 previous similar message [12051.779228] Lustre: server umount lustre-OST0000 complete [12057.057673] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12057.070308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12057.085022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12060.656099] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12060.656295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12060.679456] Lustre: Skipped 1 previous similar message [12062.658523] Lustre: server umount lustre-MDT0000 complete [12066.419731] LustreError: 305621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779710614 with bad export cookie 4499416877841272378 [12066.424880] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12066.429559] LustreError: 305621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12066.852564] Lustre: server umount lustre-MDT0001 complete [12078.883836] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [12082.005698] Key type lgssc unregistered [12085.599322] LNet: 1 local NIs in recovery (showing 1): 192.168.202.105@tcp [12086.496864] LNet: 309407:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12086.514266] LNetError: 309407:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12166.439932] LNet: Removed LNI 192.168.202.105@tcp [12167.658090] Key type .llcrypt unregistered [12167.663818] Key type ._llcrypt unregistered [12190.164684] Key type ._llcrypt registered [12190.168657] Key type .llcrypt registered [12190.271508] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [12207.068756] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [12208.404612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12208.461319] alg: No test for adler32 (adler32-zlib) [12209.460437] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [12209.650227] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [12211.319173] Key type lgssc registered [12212.800645] Lustre: Echo OBD driver; http://www.lustre.org/ [12247.390437] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 08:06:32 (1779710792) [12254.711818] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12254.777597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12256.035242] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12256.071934] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12256.136370] Lustre: lustre-MDT0000: new disk, initializing [12256.215789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12256.235584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12262.546253] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [12276.000557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12276.128456] Lustre: 312598:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12276.190899] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12276.204863] Lustre: Skipped 1 previous similar message [12276.282989] Lustre: lustre-MDT0001: new disk, initializing [12276.383205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12276.440587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12276.454826] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12281.024622] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [12290.386162] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12295.116557] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12302.193350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12302.455651] Lustre: lustre-OST0000: new disk, initializing [12302.458743] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12302.463241] Lustre: 314339:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12302.541353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12308.799742] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [12312.171387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12312.194362] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12312.337689] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12317.422555] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12327.905142] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12327.906774] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12327.912829] LustreError: Skipped 1 previous similar message [12327.913720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12327.961046] Lustre: Skipped 1 previous similar message [12333.025740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12333.037101] Lustre: Skipped 1 previous similar message [12338.163733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12338.168271] Lustre: Skipped 1 previous similar message [12342.240975] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12342.597673] Lustre: server umount lustre-OST0000 complete [12348.385370] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12348.386053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12348.408398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12348.429879] Lustre: Skipped 2 previous similar messages [12353.504176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12353.507870] Lustre: Skipped 1 previous similar message [12362.207256] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12362.465193] Lustre: server umount lustre-MDT0000 complete [12367.015672] LustreError: 312589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779710914 with bad export cookie 17862251068634347361 [12367.018116] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12367.035433] LustreError: 312589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12367.309222] Lustre: server umount lustre-MDT0001 complete [12378.032234] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [12380.936477] Key type lgssc unregistered [12381.266391] LNet: 315970:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12381.272874] LNetError: 315970:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12381.286929] LNet: Removed LNI 192.168.202.105@tcp [12382.097218] Key type .llcrypt unregistered [12382.099389] Key type ._llcrypt unregistered [12409.737756] Key type ._llcrypt registered [12409.740303] Key type .llcrypt registered [12409.823566] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [12411.261990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12411.298623] alg: No test for adler32 (adler32-zlib) [12412.324468] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [12412.506640] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [12414.215385] Key type lgssc registered [12415.714713] Lustre: Echo OBD driver; http://www.lustre.org/ [12423.418619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12423.437716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12424.867803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12428.907316] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [12434.411710] LustreError: 317375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12434.427651] LustreError: 317375:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12437.140959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12437.431303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12441.863522] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [12447.687750] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12452.097984] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12460.072880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12460.571633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12467.386129] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [12472.997379] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12481.174773] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12482.744699] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12488.674066] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12488.683807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12488.714567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12490.723882] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12490.727608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12490.741872] Lustre: Skipped 1 previous similar message [12490.764293] Lustre: Skipped 2 previous similar messages [12493.063302] Lustre: server umount lustre-MDT0000 complete [12495.842633] LustreError: 319794:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12495.871863] LustreError: 319794:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12496.976780] LustreError: 317358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779711044 with bad export cookie 7968709081484511951 [12496.997296] LustreError: 317358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12497.003331] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12500.960626] LustreError: 317371:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12500.962233] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12500.985241] LustreError: 317371:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12501.012987] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12501.015363] Lustre: Skipped 1 previous similar message [12503.008643] LustreError: 317372:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12503.031771] LustreError: 317372:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12503.492751] Lustre: server umount lustre-MDT0001 complete [12507.232700] Lustre: server umount lustre-OST0000 complete [12516.108643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12520.814368] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [12524.128371] Lustre: 321102:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.105@tcp [12527.288563] Lustre: server umount MGS complete [12537.731059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12538.576394] LustreError: 321673:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12538.775855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12545.565098] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [12549.094404] LustreError: 321674:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12549.147394] LustreError: 321674:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12556.748973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12557.233782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12562.712384] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [12572.511411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12572.792568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12580.663751] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [12587.625304] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12589.187743] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12593.996464] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12595.878832] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12601.826849] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [12601.840993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12601.871496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12601.880257] Lustre: Skipped 2 previous similar messages [12603.223290] Lustre: server umount lustre-OST0000 complete [12608.486079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12608.495856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12608.518462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12613.863829] Lustre: server umount lustre-MDT0000 complete [12615.145489] LustreError: 321674:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12615.179724] LustreError: 321674:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12617.980983] LustreError: 321654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779711165 with bad export cookie 7968709081484512798 [12617.985813] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12617.995175] LustreError: 321654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12627.837804] Lustre: server umount lustre-MDT0001 complete [12642.322083] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [12646.359252] Key type lgssc unregistered [12646.713779] LNet: 324852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12646.730368] LNetError: 324852:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12647.782817] LNet: Removed LNI 192.168.202.105@tcp [12648.898636] Key type .llcrypt unregistered [12648.903701] Key type ._llcrypt unregistered [12671.395519] Key type ._llcrypt registered [12671.399910] Key type .llcrypt registered [12671.535952] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [12689.214492] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [12691.390628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12691.529361] alg: No test for adler32 (adler32-zlib) [12692.588688] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [12692.828270] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [12694.527250] Key type lgssc registered [12696.040485] Lustre: Echo OBD driver; http://www.lustre.org/ [12739.786871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12739.818791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12741.271803] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12741.310675] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12741.399972] Lustre: lustre-MDT0000: new disk, initializing [12741.518851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12741.542066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12747.370811] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [12758.640057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12758.793167] Lustre: 327982:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12758.890265] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12758.894031] Lustre: Skipped 1 previous similar message [12758.994674] Lustre: lustre-MDT0001: new disk, initializing [12759.148902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12759.213651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12759.234553] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12764.612874] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [12773.756765] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12778.569803] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12786.578555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12787.048620] Lustre: lustre-OST0000: new disk, initializing [12787.056050] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12787.080687] Lustre: 329727:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12787.223572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12793.474442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12793.491991] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12793.570593] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [12793.656721] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12801.413783] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12807.591340] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12807.876942] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12812.907559] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12813.154796] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12819.435370] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12819.436993] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12819.441856] LustreError: Skipped 1 previous similar message [12819.444946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12819.457360] Lustre: Skipped 1 previous similar message [12819.481982] Lustre: Skipped 1 previous similar message [12821.630293] Lustre: server umount lustre-OST0000 complete [12825.568337] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12825.577439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12825.589458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12829.667886] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12829.668337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12829.684783] Lustre: Skipped 1 previous similar message [12831.864623] Lustre: server umount lustre-MDT0000 complete [12836.696821] LustreError: 327973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779711384 with bad export cookie 14586074733702661648 [12836.705265] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12836.716903] LustreError: 327973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12837.145087] Lustre: server umount lustre-MDT0001 complete [12850.794316] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 08:16:35 (1779711395) [12870.593047] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [12881.834181] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [12922.748661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12922.985204] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12923.012686] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12923.092308] Lustre: lustre-MDT0000: new disk, initializing [12923.164368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12923.185779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12928.205260] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [12939.561661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12939.712523] Lustre: 334306:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12939.774530] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12939.788089] Lustre: Skipped 1 previous similar message [12939.852716] Lustre: lustre-MDT0001: new disk, initializing [12939.920352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12939.974778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12939.985641] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12944.329836] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [12952.221825] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12956.423612] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12963.055026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12963.397078] Lustre: lustre-OST0000: new disk, initializing [12963.404099] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12963.416449] Lustre: 336049:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12963.527330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12970.854191] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [12971.627836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12971.648687] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12971.806666] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12979.188837] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13023.203360] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13023.220590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13023.224702] Lustre: Skipped 1 previous similar message [13023.229965] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13023.232499] Lustre: Skipped 1 previous similar message [13026.876064] Lustre: server umount lustre-OST0000 complete [13032.420811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13032.431625] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13032.453960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13037.458533] Lustre: server umount lustre-MDT0000 complete [13038.568216] LustreError: 334319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13038.591349] LustreError: 334319:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [13041.306702] LustreError: 334299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779711589 with bad export cookie 14586074733702662530 [13041.326447] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13041.781061] Lustre: server umount lustre-MDT0001 complete [13053.839074] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [13056.505867] Key type lgssc unregistered [13056.767823] LNet: 338093:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13056.773108] LNetError: 338093:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13057.832391] LNet: Removed LNI 192.168.202.105@tcp [13058.844529] Key type .llcrypt unregistered [13058.853464] Key type ._llcrypt unregistered [13169.290770] Key type ._llcrypt registered [13169.293249] Key type .llcrypt registered [13169.430850] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [13169.810613] Key type .llcrypt unregistered [13169.818483] Key type ._llcrypt unregistered [13263.771317] Key type ._llcrypt registered [13263.778271] Key type .llcrypt registered [13263.949161] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [13264.318512] Key type .llcrypt unregistered [13264.329540] Key type ._llcrypt unregistered [13306.028775] Key type ._llcrypt registered [13306.031949] Key type .llcrypt registered [13306.174293] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [13306.604438] Key type .llcrypt unregistered [13306.606912] Key type ._llcrypt unregistered [13346.156635] Key type ._llcrypt registered [13346.167724] Key type .llcrypt registered [13346.420937] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [13346.839492] Key type .llcrypt unregistered [13346.847914] Key type ._llcrypt unregistered [13448.299720] Key type ._llcrypt registered [13448.305605] Key type .llcrypt registered [13448.428904] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [13448.780345] Key type .llcrypt unregistered [13448.782152] Key type ._llcrypt unregistered [13485.736435] Key type ._llcrypt registered [13485.739460] Key type .llcrypt registered [13485.846427] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [13486.224340] Key type .llcrypt unregistered [13486.226114] Key type ._llcrypt unregistered [13523.283712] Key type ._llcrypt registered [13523.288716] Key type .llcrypt registered [13523.428235] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [13523.836458] Key type .llcrypt unregistered [13523.839695] Key type ._llcrypt unregistered [13560.677621] Key type ._llcrypt registered [13560.680576] Key type .llcrypt registered [13560.823189] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [13561.395543] Key type .llcrypt unregistered [13561.407787] Key type ._llcrypt unregistered [13605.162727] Key type ._llcrypt registered [13605.165067] Key type .llcrypt registered [13605.305250] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [13605.747555] Key type .llcrypt unregistered [13605.756259] Key type ._llcrypt unregistered [13642.805531] Key type ._llcrypt registered [13642.810995] Key type .llcrypt registered [13642.936324] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [13643.214597] Key type .llcrypt unregistered [13643.217503] Key type ._llcrypt unregistered [13716.682571] Key type ._llcrypt registered [13716.685627] Key type .llcrypt registered [13716.856328] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [13717.152192] Key type .llcrypt unregistered [13717.153871] Key type ._llcrypt unregistered [13752.593638] Key type ._llcrypt registered [13752.598461] Key type .llcrypt registered [13752.771532] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [13753.177777] Key type .llcrypt unregistered [13753.183838] Key type ._llcrypt unregistered [13790.414379] Key type ._llcrypt registered [13790.417534] Key type .llcrypt registered [13790.517687] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [13790.885588] Key type .llcrypt unregistered [13790.890791] Key type ._llcrypt unregistered [13848.287546] Key type ._llcrypt registered [13848.292922] Key type .llcrypt registered [13848.450758] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [13865.793181] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [13867.114081] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13867.145596] alg: No test for adler32 (adler32-zlib) [13868.457248] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [13868.661613] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [13870.351217] Key type lgssc registered [13871.623576] Lustre: Echo OBD driver; http://www.lustre.org/ [13911.953718] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13912.025907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13913.558203] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13913.645289] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13913.795376] Lustre: tst250fs-MDT0000: new disk, initializing [13913.947500] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13913.973442] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13918.386911] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [13930.492301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13930.657846] Lustre: 358225:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13930.742062] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13930.750741] Lustre: Skipped 1 previous similar message [13930.820654] Lustre: tst250fs-MDT0001: new disk, initializing [13930.947390] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13931.003887] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13931.020112] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13936.770918] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [13942.962737] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13952.025626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13952.420920] Lustre: tst250fs-OST0000: new disk, initializing [13952.434214] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13952.448961] Lustre: 359820:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13952.664997] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13958.247244] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13958.267576] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13958.399717] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13960.144835] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [13971.929517] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13972.166413] Lustre: tst250fs-OST0001: new disk, initializing [13972.171831] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13972.175674] Lustre: 360676:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13972.293064] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13978.750720] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [13980.229583] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13980.242135] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13980.355659] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13984.602187] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13987.473842] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13987.477229] Lustre: Skipped 2 previous similar messages [14007.780949] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14007.789941] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14007.810189] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14011.364318] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14011.366578] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14011.408075] Lustre: Skipped 1 previous similar message [14013.788324] Lustre: server umount tst250fs-MDT0000 complete [14018.089837] LustreError: 358216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779712565 with bad export cookie 13769702001192355046 [14018.094161] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14018.102891] LustreError: 358216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14018.590536] Lustre: server umount tst250fs-MDT0001 complete [14032.312684] Lustre: server umount tst250fs-OST0000 complete [14045.569529] Lustre: server umount tst250fs-OST0001 complete [14065.516402] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [14076.729094] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [14117.663464] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 08:37:42 (1779712662) [14120.636224] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14144.510266] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [14155.122604] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [14191.241900] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13933 sec ========= 08:38:56 (1779712736) [14194.601234] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:38:59 (1779712739) === [14199.003212] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:39:04 (1779712744) === [14246.563457] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [14249.424504] Key type lgssc unregistered [14249.802892] LNet: 369724:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14249.808097] LNetError: 369724:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14249.832201] LNet: Removed LNI 192.168.202.105@tcp [14250.810257] Key type .llcrypt unregistered [14250.815397] Key type ._llcrypt unregistered