[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 468512506 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003268] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.007753] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010149] LSM: Security Framework initializing [ 0.011052] Yama: becoming mindful. [ 0.012040] SELinux: Initializing. [ 0.013068] *** VALIDATE selinux *** [ 0.021842] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025694] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026167] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027125] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028141] *** VALIDATE tmpfs *** [ 0.030315] *** VALIDATE proc *** [ 0.031235] *** VALIDATE cgroup *** [ 0.032009] *** VALIDATE cgroup2 *** [ 0.033253] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034159] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036030] Spectre V2 : User space: Vulnerable [ 0.037009] Speculative Store Bypass: Vulnerable [ 0.040257] debug: unmapping init [mem 0xffffffffbe059000-0xffffffffbe060fff] [ 0.043000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043694] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044025] ... version: 2 [ 0.045012] ... bit width: 48 [ 0.046011] ... generic registers: 4 [ 0.047012] ... value mask: 0000ffffffffffff [ 0.048012] ... max period: 00007fffffffffff [ 0.049017] ... fixed-purpose events: 3 [ 0.050014] ... event mask: 000000070000000f [ 0.051293] rcu: Hierarchical SRCU implementation. [ 0.053473] smp: Bringing up secondary CPUs ... [ 0.054609] x86: Booting SMP configuration: [ 0.055025] .... node #0, CPUs: #1 #2 #3 [ 0.062083] smp: Brought up 1 node, 4 CPUs [ 0.064014] smpboot: Max logical packages: 1 [ 0.065015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.124029] node 0 deferred pages initialised in 57ms [ 0.127472] devtmpfs: initialized [ 0.128250] x86/mm: Memory block size: 128MB [ 0.130957] gcov: version magic: 0x41383552 [ 0.135162] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.139105] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.142348] pinctrl core: initialized pinctrl subsystem [ 0.144176] [ 0.144907] ************************************************************* [ 0.148014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.150010] ** ** [ 0.153014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.161016] ** ** [ 0.163012] ** This means that this kernel is built to expose internal ** [ 0.166012] ** IOMMU data structures, which may compromise security on ** [ 0.169011] ** your system. ** [ 0.171013] ** ** [ 0.173016] ** If you see this message and you are not debugging the ** [ 0.176014] ** kernel, report this immediately to your vendor! ** [ 0.178017] ** ** [ 0.181011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.183012] ************************************************************* [ 0.186129] NET: Registered protocol family 16 [ 0.188575] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.191134] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.194070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.197103] cpuidle: using governor menu [ 0.199638] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.202502] PCI: Using configuration type 1 for base access [ 0.205135] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.212010] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.213015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.214146] cryptd: max_cpu_qlen set to 1000 [ 0.217195] ACPI: Added _OSI(Module Device) [ 0.218009] ACPI: Added _OSI(Processor Device) [ 0.218994] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.220014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.224360] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.229464] ACPI: Interpreter enabled [ 0.230106] ACPI: PM: (supports S0 S3 S4 S5) [ 0.230976] ACPI: Using IOAPIC for interrupt routing [ 0.232157] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.235468] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.243000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.246044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.249019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.252094] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.257464] acpiphp: Slot [2] registered [ 0.259155] acpiphp: Slot [5] registered [ 0.261127] acpiphp: Slot [6] registered [ 0.262106] acpiphp: Slot [7] registered [ 0.264118] acpiphp: Slot [8] registered [ 0.265158] acpiphp: Slot [9] registered [ 0.267061] acpiphp: Slot [10] registered [ 0.268114] acpiphp: Slot [3] registered [ 0.269062] acpiphp: Slot [4] registered [ 0.270043] acpiphp: Slot [11] registered [ 0.271083] acpiphp: Slot [12] registered [ 0.272034] acpiphp: Slot [13] registered [ 0.272978] acpiphp: Slot [14] registered [ 0.274081] acpiphp: Slot [15] registered [ 0.275150] acpiphp: Slot [16] registered [ 0.277130] acpiphp: Slot [17] registered [ 0.278115] acpiphp: Slot [18] registered [ 0.280103] acpiphp: Slot [19] registered [ 0.281208] acpiphp: Slot [20] registered [ 0.283088] acpiphp: Slot [21] registered [ 0.284096] acpiphp: Slot [22] registered [ 0.285121] acpiphp: Slot [23] registered [ 0.286089] acpiphp: Slot [24] registered [ 0.286986] acpiphp: Slot [25] registered [ 0.288066] acpiphp: Slot [26] registered [ 0.289029] acpiphp: Slot [27] registered [ 0.290000] acpiphp: Slot [28] registered [ 0.290971] acpiphp: Slot [29] registered [ 0.292087] acpiphp: Slot [30] registered [ 0.293044] acpiphp: Slot [31] registered [ 0.294073] PCI host bridge to bus 0000:00 [ 0.295014] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.297015] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.298014] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.300024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.301015] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.303024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.304131] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.305000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.306000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.315561] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.320043] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.322014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.324026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.327020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.330436] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.332870] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.335039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.337890] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.342013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.354017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.357911] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.362323] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.367014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.373018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.385014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.394637] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.401022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.406018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.426019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.433000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.442015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.451019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.470023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.477000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.488017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.494014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.511016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.520272] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.526015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.533018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.551027] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.564490] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.570016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.575013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.593016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.602173] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.604430] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.606337] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.608336] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.610148] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.614178] iommu: Default domain type: Passthrough [ 0.615498] SCSI subsystem initialized [ 0.617097] ACPI: bus type USB registered [ 0.618070] usbcore: registered new interface driver usbfs [ 0.619058] usbcore: registered new interface driver hub [ 0.621075] usbcore: registered new device driver usb [ 0.622160] pps_core: LinuxPPS API ver. 1 registered [ 0.623007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.625039] PTP clock support registered [ 0.627103] EDAC MC: Ver: 3.0.0 [ 0.628534] PCI: Using ACPI for IRQ routing [ 0.630772] NetLabel: Initializing [ 0.631008] NetLabel: domain hash size = 128 [ 0.632007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.633068] NetLabel: unlabeled traffic allowed by default [ 0.635098] vgaarb: loaded [ 0.636185] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.637006] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.643268] clocksource: Switched to clocksource kvm-clock [ 0.746494] VFS: Disk quotas dquot_6.6.0 [ 0.747627] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.749906] *** VALIDATE ramfs *** [ 0.750749] *** VALIDATE hugetlbfs *** [ 0.751762] pnp: PnP ACPI init [ 0.753648] pnp: PnP ACPI: found 6 devices [ 0.770134] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.773600] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.775099] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.776603] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.779032] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.781684] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.784572] NET: Registered protocol family 2 [ 0.786843] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.792123] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.795676] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.801603] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.805388] TCP: Hash tables configured (established 65536 bind 65536) [ 0.808519] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.811806] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.814743] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.816895] NET: Registered protocol family 1 [ 0.819544] RPC: Registered named UNIX socket transport module. [ 0.822023] RPC: Registered udp transport module. [ 0.823846] RPC: Registered tcp transport module. [ 0.825691] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.828038] NET: Registered protocol family 44 [ 0.829715] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.831995] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.834201] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.836692] PCI: CLS 0 bytes, default 64 [ 0.838990] Unpacking initramfs... [ 2.257119] debug: unmapping init [mem 0xffff8e1c3cc54000-0xffff8e1c3ffbffff] [ 2.259756] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.261059] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.262827] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.779130] Initialise system trusted keyrings [ 2.780778] Key type blacklist registered [ 2.783069] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.792040] zbud: loaded [ 2.794906] *** VALIDATE nfs *** [ 2.796044] *** VALIDATE nfs4 *** [ 2.797511] pstore: using deflate compression [ 2.801833] Platform Keyring initialized [ 2.927548] NET: Registered protocol family 38 [ 2.929594] Key type asymmetric registered [ 2.932195] Asymmetric key parser 'x509' registered [ 2.934159] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.937093] io scheduler mq-deadline registered [ 2.938538] io scheduler kyber registered [ 2.940106] io scheduler bfq registered [ 2.942827] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.946550] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.949361] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.952378] ACPI: Power Button [PWRF] [ 2.958148] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.964694] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.986514] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.996535] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.012195] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.040290] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.071161] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.075900] Non-volatile memory driver v1.3 [ 3.077743] Linux agpgart interface v0.103 [ 3.106438] virtio_blk virtio1: [vda] 146152 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.109165] vda: detected capacity change from 0 to 74829824 [ 3.126293] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.129897] vdb: detected capacity change from 0 to 1073741824 [ 3.147670] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.151156] vdc: detected capacity change from 0 to 2621440000 [ 3.166460] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.169814] vdd: detected capacity change from 0 to 2621440000 [ 3.183966] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.187403] vde: detected capacity change from 0 to 4294967296 [ 3.202675] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.205991] vdf: detected capacity change from 0 to 4294967296 [ 3.219203] libphy: Fixed MDIO Bus: probed [ 3.242371] usbcore: registered new interface driver usbserial_generic [ 3.245182] usbserial: USB Serial support registered for generic [ 3.247758] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.253139] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.255391] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.257953] mousedev: PS/2 mouse device common for all mice [ 3.261399] rtc_cmos 00:05: RTC can wake from S4 [ 3.264623] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.265434] rtc_cmos 00:05: registered as rtc0 [ 3.270633] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.274071] intel_pstate: CPU model not supported [ 3.277038] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.281593] hid: raw HID events driver (C) Jiri Kosina [ 3.283962] usbcore: registered new interface driver usbhid [ 3.286095] usbhid: USB HID core driver [ 3.287963] drop_monitor: Initializing network drop monitor service [ 3.290886] Initializing XFRM netlink socket [ 3.291197] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.293101] NET: Registered protocol family 10 [ 3.299459] Segment Routing with IPv6 [ 3.301101] NET: Registered protocol family 17 [ 3.303438] mpls_gso: MPLS GSO support [ 3.309593] RAS: Correctable Errors collector initialized. [ 3.311821] AVX version of gcm_enc/dec engaged. [ 3.313729] AES CTR mode by8 optimization enabled [ 3.399862] sched_clock: Marking stable (3399842547, 0)->(4288112698, -888270151) [ 3.403374] registered taskstats version 1 [ 3.405399] Loading compiled-in X.509 certificates [ 3.407426] zswap: loaded using pool lzo/zbud [ 3.429088] Key type big_key registered [ 3.441658] Key type encrypted registered [ 3.442804] ima: No TPM chip found, activating TPM-bypass! [ 3.444039] ima: Allocated hash algorithm: sha1 [ 3.445301] ima: No architecture policies found [ 3.446350] evm: Initialising EVM extended attributes: [ 3.447532] evm: security.selinux [ 3.448316] evm: security.ima [ 3.449195] evm: security.capability [ 3.450097] evm: HMAC attrs: 0x1 [ 3.452442] rtc_cmos 00:05: setting system clock to 2026-08-24 17:43:55 UTC (1787593435) [ 3.457380] debug: unmapping init [mem 0xffffffffbf003000-0xffffffffbf1fffff] [ 3.459132] debug: unmapping init [mem 0xffffffffbdd82000-0xffffffffbe058fff] [ 3.467068] Write protecting the kernel read-only data: 28672k [ 3.470752] debug: unmapping init [mem 0xffffffffbc403000-0xffffffffbc5fffff] [ 3.473697] debug: unmapping init [mem 0xffffffffbcd14000-0xffffffffbcdfffff] [ 3.503774] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.511193] systemd[1]: Detected virtualization kvm. [ 3.512695] systemd[1]: Detected architecture x86-64. [ 3.513872] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.545071] systemd[1]: No hostname configured. [ 3.546539] systemd[1]: Set hostname to . [ 3.548892] random: systemd: uninitialized urandom read (16 bytes read) [ 3.551704] systemd[1]: Initializing machine ID from random generator. [ 3.603925] random: ln: uninitialized urandom read (6 bytes read) [ 3.697959] random: systemd: uninitialized urandom read (16 bytes read) [ 3.701292] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.707464] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ 3.712908] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.476695] device-mapper: uevent: version 1.0.3 [ 4.479663] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.415451] virtio_net virtio0 ens2: renamed from eth0 [ 5.439824] random: fast init done [ 5.475781] scsi host0: ata_piix [ 5.526865] scsi host1: ata_piix [ 5.553829] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.556834] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.794592] dracut-initqueue[585]: RTNETLINK answers: File exists [ 9.972463] random: crng init done [ 9.979360] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 12.686246] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.071395] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.238260] SELinux: Disabled at runtime. [ 18.452632] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.474679] systemd[1]: Detected virtualization kvm. [ 18.482667] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.057051] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.070593] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.085276] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.095595] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.116365] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.172747] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.225394] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Created slice User and Session Slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. [ 21.860410] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 23.424658] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 24.708287] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.811510] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.423992] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 25.535532] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit)[ 32.396106] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (11s / no limit)[ 32.930109] hrtimer: interrupt took 2940387 ns [ **] A start job is running for Configur…only root support (12s / no limit) [ *] A start job [ 34.010759] NFS: Registering the id_resolver key type [ 34.038123] Key type id_resolver registered is running for Configur…only root support (13s / no limit)[ 34.061516] Key type id_legacy registered [ **] A start job is running for Configur…only root support (13s / no limit) [ ***] A start job is running for Configur…only root support (14s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg259-server login: [ 97.533873] libcfs: loading out-of-tree module taints kernel. [ 97.571307] Key type ._llcrypt registered [ 97.575208] Key type .llcrypt registered [ 97.655808] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 119.197698] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 121.390569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 121.412311] alg: No test for adler32 (adler32-zlib) [ 123.460742] Lustre: Lustre: Build Version: 2.17.57_80_g3abb7b2 [ 124.662496] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 126.544186] Key type lgssc registered [ 129.018543] Lustre: Echo OBD driver; http://www.lustre.org/ [ 150.389067] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 200.470243] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 215.186406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 215.229920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 216.566602] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 216.598884] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 216.699591] Lustre: lustre-MDT0000: new disk, initializing [ 216.787703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 216.819521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 221.757502] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.980561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 238.164496] Lustre: 6521:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 238.215886] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 238.220571] Lustre: Skipped 1 previous similar message [ 238.346435] Lustre: lustre-MDT0001: new disk, initializing [ 238.469658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 238.517313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 238.542431] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 244.674689] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 250.535463] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 261.056857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 261.474961] Lustre: lustre-OST0000: new disk, initializing [ 261.491191] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 261.499879] Lustre: 8458:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 261.646619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 266.281191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 266.297272] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 266.455943] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 268.973750] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 288.597679] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 288.846093] Lustre: lustre-OST0001: new disk, initializing [ 288.851117] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 288.866161] Lustre: 9531:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 288.951606] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 295.482079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 295.495319] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 295.558039] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 296.554915] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 310.962705] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 316.261896] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 323.592706] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing check_logdir /tmp/testlogs/ [ 330.822563] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing yml_node [ 335.769885] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 338.854937] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 341.631925] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 343.790678] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 24 13:49:34 EDT 2026 [ 363.300737] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 365.160603] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 385.505038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 385.506764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 385.521601] Lustre: Skipped 1 previous similar message [ 385.535301] Lustre: Skipped 3 previous similar messages [ 390.743662] Lustre: server umount lustre-MDT0000 complete [ 395.007108] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787593827 with bad export cookie 2138919203989653921 [ 395.013700] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 395.023600] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 395.745915] LustreError: 6528:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 395.748993] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 395.749643] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 395.780919] LustreError: 6528:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 395.800741] Lustre: Skipped 3 previous similar messages [ 400.867803] LustreError: 6527:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 400.870882] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 400.896768] Lustre: Skipped 2 previous similar messages [ 401.851369] Lustre: server umount lustre-MDT0001 complete [ 407.521969] Lustre: server umount lustre-OST0000 complete [ 412.997936] Lustre: server umount lustre-OST0001 complete [ 423.288112] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 439.886127] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 481.683714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 482.009746] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 482.058690] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 482.207703] Lustre: lustre-MDT0000: new disk, initializing [ 482.327795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 482.351379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 488.302665] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 500.954301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 501.187417] Lustre: 15108:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 501.301432] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 501.324371] Lustre: Skipped 1 previous similar message [ 501.483098] Lustre: lustre-MDT0001: new disk, initializing [ 501.666472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 501.719680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 501.733881] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 507.291987] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 521.226723] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 528.814053] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 536.485671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 536.860322] Lustre: lustre-OST0000: new disk, initializing [ 536.864720] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 536.871496] Lustre: 17080:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 536.982453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 538.597395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 538.611516] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 538.733540] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 543.640521] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 556.627534] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 562.128976] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 562.346042] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 568.186413] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 568.570835] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 574.436520] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 574.454215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 574.483653] Lustre: Skipped 1 previous similar message [ 574.525795] Lustre: Skipped 1 previous similar message [ 578.907871] Lustre: server umount lustre-OST0000 complete [ 588.784138] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 588.796289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 588.828978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 591.699974] Lustre: server umount lustre-MDT0000 complete [ 594.913592] LustreError: 15963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 594.940619] LustreError: 15963:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 597.096139] LustreError: 15100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787594029 with bad export cookie 2138919203989655447 [ 597.099430] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 597.115931] LustreError: 15100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 597.785672] Lustre: server umount lustre-MDT0001 complete [ 605.135839] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 13:53:55 (1787594035) [ 627.268792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 628.009343] LustreError: 19958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 628.036609] LustreError: 19958:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 628.116836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 633.306266] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 633.316886] LustreError: 19959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 643.561552] LustreError: 19958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 643.598768] LustreError: 19958:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 643.641348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 650.687943] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 661.926749] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 669.930792] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 673.238268] Lustre: Failing over lustre-MDT0000 [ 673.824825] Lustre: server umount lustre-MDT0000 complete [ 675.311496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 675.322533] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 675.341286] Lustre: Skipped 1 previous similar message [ 675.347147] LustreError: 19959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 694.700707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 694.752188] Lustre: 3652:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787594110/real 1787594110] req@ffff8e1b82cf9880 x1874427700751232/t0(0) o400->MGC192.168.202.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787594126 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 694.805102] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 703.974349] LustreError: 3649:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8e1b82cf9c00 x1874427700753792/t0(0) o250->MGC192.168.202.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 704.511336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 704.524489] Lustre: Skipped 1 previous similar message [ 704.601766] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 709.616956] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 709.631669] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 709.662498] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 709.713341] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 720.796280] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 725.539279] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 725.547075] Lustre: Skipped 1 previous similar message [ 725.550218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 725.553256] Lustre: Skipped 2 previous similar messages [ 732.054594] Lustre: server umount lustre-MDT0000 complete [ 735.719771] LustreError: 19959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 735.743546] LustreError: 19959:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 736.894239] LustreError: 19940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787594168 with bad export cookie 2138919203989656868 [ 736.903154] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 736.918201] LustreError: 19940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 737.444360] Lustre: server umount lustre-MDT0001 complete [ 748.578124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 755.074686] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 765.591917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 772.121738] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 783.989943] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 794.053898] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 796.711565] Lustre: Failing over lustre-MDT0000 [ 797.170852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 797.185413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 797.202873] Lustre: Skipped 1 previous similar message [ 797.249800] Lustre: server umount lustre-MDT0000 complete [ 804.344506] LustreError: 23299:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 804.364672] LustreError: 23299:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 815.586000] Lustre: 3650:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787594231/real 1787594231] req@ffff8e1c812a9500 x1874427700784128/t0(0) o400->MGC192.168.202.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787594247 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 815.623200] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 818.514829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 825.826616] LustreError: 3649:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8e1ca43e2d80 x1874427700786816/t0(0) o250->MGC192.168.202.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 826.132688] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 831.006222] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 831.463299] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 831.464330] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 831.472559] Lustre: Skipped 1 previous similar message [ 831.520385] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 842.179905] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 846.837145] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 846.857484] Lustre: Skipped 1 previous similar message [ 846.870736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 846.879427] Lustre: Skipped 3 previous similar messages [ 851.939311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 853.354312] Lustre: server umount lustre-MDT0000 complete [ 858.159964] LustreError: 23280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787594290 with bad export cookie 2138919203989657806 [ 858.180743] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 858.190092] LustreError: 23280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 858.650402] Lustre: server umount lustre-MDT0001 complete [ 869.778412] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 13:58:20 (1787594300) [ 889.894375] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 902.785489] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 950.700936] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 966.075070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 966.590524] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 966.688517] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 966.888571] Lustre: lustre-MDT0000: new disk, initializing [ 967.073576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 967.079635] Lustre: Skipped 3 previous similar messages [ 967.103610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 973.473872] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 984.665460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 984.805926] Lustre: 29686:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 985.154519] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 990.145978] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1001.266283] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1008.564629] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1016.647539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1016.932714] Lustre: 31656:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1018.514714] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1018.585036] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 1018.691566] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1018.716788] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 1025.492559] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1037.968938] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1044.495307] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1070.049779] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1070.053208] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1070.059402] LustreError: Skipped 1 previous similar message [ 1070.084790] Lustre: Skipped 1 previous similar message [ 1070.087108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1070.109776] Lustre: Skipped 3 previous similar messages [ 1082.338954] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1082.546603] Lustre: server umount lustre-OST0000 complete [ 1087.459578] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1093.137308] Lustre: server umount lustre-MDT0000 complete [ 1095.670377] LustreError: 30533:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1095.706961] LustreError: 30533:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 1097.660528] LustreError: 29678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787594529 with bad export cookie 2138919203989658212 [ 1097.665140] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1097.672356] LustreError: 29678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1098.264218] Lustre: server umount lustre-MDT0001 complete [ 1111.831357] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1115.394714] Key type lgssc unregistered [ 1115.760369] LNet: 33651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1115.772428] LNetError: 33651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1116.849194] LNet: Removed LNI 192.168.202.159@tcp [ 1118.112406] Key type .llcrypt unregistered [ 1118.114933] Key type ._llcrypt unregistered [ 1145.795812] Key type ._llcrypt registered [ 1145.799103] Key type .llcrypt registered [ 1145.901422] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1147.228758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1147.280446] alg: No test for adler32 (adler32-zlib) [ 1148.394575] Lustre: Lustre: Build Version: 2.17.57_80_g3abb7b2 [ 1148.703899] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1150.424364] Key type lgssc registered [ 1151.736256] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1159.678156] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1159.698894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1161.170555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1166.581350] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1171.428342] LustreError: 35067:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1171.448093] LustreError: 35067:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1176.398839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1176.547633] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1176.766445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1181.958703] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1190.179567] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1196.155801] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1203.602478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1203.995717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1206.102687] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1210.484263] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1219.074988] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1221.486371] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1237.318158] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 14:04:27 (1787594667) [ 1239.648446] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1242.066346] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1250.847774] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 14:04:40 (1787594680) [ 1260.228268] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 14:04:50 (1787594690) [ 1261.574103] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1270.109074] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1270.117702] Lustre: Skipped 2 previous similar messages [ 1271.825846] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 14:05:02 (1787594702) [ 1275.597965] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1275.602510] Lustre: Skipped 3 previous similar messages [ 1294.575260] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1294.580827] Lustre: Skipped 6 previous similar messages [ 1297.713845] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 14:05:28 (1787594728) [ 1301.194107] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1305.285636] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1314.746438] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 14:05:45 (1787594745) [ 1316.628661] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1316.634197] Lustre: Skipped 3 previous similar messages [ 1330.594139] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 14:06:00 (1787594760) [ 1349.605143] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1349.608946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1349.610543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1349.622637] Lustre: Skipped 1 previous similar message [ 1351.932574] Lustre: server umount lustre-OST0000 complete [ 1356.259810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1356.268640] LustreError: Skipped 1 previous similar message [ 1356.278872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1356.293179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1356.298059] Lustre: Skipped 1 previous similar message [ 1359.841116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1359.846263] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1362.449104] Lustre: server umount lustre-MDT0000 complete [ 1364.965327] LustreError: 35066:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1364.976667] LustreError: 35066:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1366.398787] LustreError: 35047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787594798 with bad export cookie 976368788093722129 [ 1366.400454] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1366.411595] LustreError: 35047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1366.885615] Lustre: server umount lustre-MDT0001 complete [ 1380.211716] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1383.317256] Key type lgssc unregistered [ 1383.686768] LNet: 42938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1383.692901] LNetError: 42938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1383.703815] LNet: Removed LNI 192.168.202.159@tcp [ 1384.614501] Key type .llcrypt unregistered [ 1384.617978] Key type ._llcrypt unregistered [ 1404.981476] Key type ._llcrypt registered [ 1404.987285] Key type .llcrypt registered [ 1405.116949] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 1424.008515] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1425.613626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1425.664522] alg: No test for adler32 (adler32-zlib) [ 1426.936938] Lustre: Lustre: Build Version: 2.17.57_80_g3abb7b2 [ 1427.238064] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1428.931391] Key type lgssc registered [ 1430.380910] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1473.133842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1473.160574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1474.463789] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1474.491596] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1474.652123] Lustre: lustre-MDT0000: new disk, initializing [ 1474.765038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1474.796369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1481.945258] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1497.881604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1498.075838] Lustre: 46094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1498.215876] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1498.222318] Lustre: Skipped 1 previous similar message [ 1498.348195] Lustre: lustre-MDT0001: new disk, initializing [ 1498.558124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1498.607539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1498.635466] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1504.528888] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1515.870824] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1521.951106] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1529.688916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1529.977434] Lustre: lustre-OST0000: new disk, initializing [ 1529.986873] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1530.004695] Lustre: 48066:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1530.107584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1537.104693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1537.120339] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1537.255417] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1537.697378] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1547.967671] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1553.556155] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1553.793560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1559.661335] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1559.942892] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1563.117659] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1563.124464] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1563.152019] LustreError: Skipped 1 previous similar message [ 1563.164091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1563.182666] Lustre: Skipped 1 previous similar message [ 1563.221726] Lustre: Skipped 1 previous similar message [ 1568.232107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1568.244987] Lustre: Skipped 1 previous similar message [ 1569.101467] Lustre: server umount lustre-OST0000 complete [ 1575.393894] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1575.399811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1575.430783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1578.473319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1578.478715] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1582.438392] Lustre: server umount lustre-MDT0000 complete [ 1586.978102] LustreError: 46088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787595019 with bad export cookie 9890148560320664011 [ 1586.991499] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1586.993645] LustreError: 46088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1587.531032] Lustre: server umount lustre-MDT0001 complete [ 1598.807362] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 14:10:29 (1787595029) [ 1606.761391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1607.207988] LustreError: 50538:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1607.233451] LustreError: 50538:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1607.301417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1612.270206] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1612.780508] LustreError: 50539:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1617.894449] LustreError: 50538:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1621.831469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1622.432667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1627.443761] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1636.456795] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1643.549119] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1650.841336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1651.242386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1658.704712] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1666.337534] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1671.117763] LustreError: 53066:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1672.250341] Lustre: Modifying parameter general.timeout=1 in log params [ 1673.359047] Lustre: Modifying parameter general.timeout=2 in log params [ 1675.874639] Lustre: Modifying parameter general.timeout=5 in log params [ 1675.882403] Lustre: Skipped 2 previous similar messages [ 1680.402750] Lustre: Modifying parameter general.timeout=10 in log params [ 1680.407423] Lustre: Skipped 4 previous similar messages [ 1689.200660] Lustre: Modifying parameter general.timeout=20 in log params [ 1689.204092] Lustre: Skipped 9 previous similar messages [ 1705.376258] Lustre: Modifying parameter general.timeout=36 in log params [ 1705.378915] Lustre: Skipped 15 previous similar messages [ 1738.003426] Lustre: Modifying parameter general.timeout=69 in log params [ 1738.010665] Lustre: Skipped 32 previous similar messages [ 1801.761414] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 14:13:52 (1787595232) [ 1802.853533] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1802.863276] Lustre: Skipped 61 previous similar messages [ 1804.884933] Lustre: 60006:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1806.676392] Lustre: 60104:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1810.062950] Lustre: 60317:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1810.072709] Lustre: 60317:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1843.328101] Lustre: 61873:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1843.342165] Lustre: 61873:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1843.369272] LustreError: 61873:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1848.084488] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 14:14:38 (1787595278) [ 1861.603084] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1861.605524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1861.627624] Lustre: Skipped 1 previous similar message [ 1861.632417] Lustre: Skipped 1 previous similar message [ 1866.724272] LustreError: 50538:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1866.740069] LustreError: 50538:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1867.004180] Lustre: server umount lustre-MDT0000 complete [ 1870.542644] LustreError: 50519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787595302 with bad export cookie 9890148560320664893 [ 1870.555212] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1870.558366] LustreError: 50519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1870.980378] Lustre: server umount lustre-MDT0001 complete [ 1885.155419] Lustre: server umount lustre-OST0000 complete [ 1913.855852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1914.163583] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1914.627412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1919.479430] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1928.867430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1928.952876] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1928.985696] Lustre: 64406:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1934.845332] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1943.829468] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1950.150385] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1957.824863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1958.036521] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1958.346161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1958.357252] Lustre: Skipped 1 previous similar message [ 1964.464465] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1972.745745] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1974.922987] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1974.930721] Lustre: Skipped 22 previous similar messages [ 1975.035411] Lustre: 66971:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1986.540297] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1986.541306] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1986.545392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1986.558472] Lustre: Skipped 2 previous similar messages [ 1986.584933] Lustre: Skipped 3 previous similar messages [ 1988.872488] Lustre: server umount lustre-OST0000 complete [ 1995.747337] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1995.754876] LustreError: Skipped 1 previous similar message [ 1995.758537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1999.316395] Lustre: server umount lustre-MDT0000 complete [ 2001.891149] LustreError: 64411:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2001.914102] LustreError: 64411:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2002.880045] LustreError: 64397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787595434 with bad export cookie 9890148560320667910 [ 2002.892425] LustreError: 64397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2002.896144] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2003.292406] Lustre: server umount lustre-MDT0001 complete [ 2016.662612] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 2019.793910] Key type lgssc unregistered [ 2020.099149] LNet: 68186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2020.111758] LNetError: 68186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2021.162149] LNet: Removed LNI 192.168.202.159@tcp [ 2022.132486] Key type .llcrypt unregistered [ 2022.135258] Key type ._llcrypt unregistered [ 2051.077241] Key type ._llcrypt registered [ 2051.079630] Key type .llcrypt registered [ 2051.176429] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2052.302800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2052.333907] alg: No test for adler32 (adler32-zlib) [ 2053.360723] Lustre: Lustre: Build Version: 2.17.57_80_g3abb7b2 [ 2053.556788] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 2055.216213] Key type lgssc registered [ 2056.239309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2064.185750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2064.218229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2065.808088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2070.389339] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2075.106853] LustreError: 69602:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2075.133523] LustreError: 69602:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2078.797427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2079.545910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2084.116779] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2093.072840] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2100.149613] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2108.714117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2109.271313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2116.295728] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2125.513642] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2133.902616] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2145.761601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2145.770647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2145.789954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2146.281180] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2146.294927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2146.299632] Lustre: Skipped 1 previous similar message [ 2146.312311] Lustre: Skipped 2 previous similar messages [ 2150.234324] Lustre: server umount lustre-MDT0000 complete [ 2151.405297] LustreError: 69597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2151.430156] LustreError: 69597:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2154.414929] LustreError: 69582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787595586 with bad export cookie 135275076756573400 [ 2154.416569] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2154.426875] LustreError: 69582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2154.675770] Lustre: server umount lustre-MDT0001 complete [ 2169.019889] Lustre: server umount lustre-OST0000 complete [ 2195.167067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2195.501489] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2195.543799] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2195.805052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2199.847639] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2209.045697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2209.160175] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2209.177102] Lustre: 74473:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2209.565832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2215.127922] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2224.222807] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2229.934759] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2237.231028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2237.401485] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2237.788519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2244.000890] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2251.908849] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2253.776395] Lustre: 77050:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2255.748338] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2264.040387] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2264.043223] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2264.069871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2264.074019] LustreError: Skipped 1 previous similar message [ 2264.088601] Lustre: Skipped 1 previous similar message [ 2267.769564] Lustre: server umount lustre-OST0000 complete [ 2274.274815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2274.280367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2274.294744] Lustre: Skipped 2 previous similar messages [ 2274.317270] Lustre: Skipped 1 previous similar message [ 2279.047952] Lustre: server umount lustre-MDT0000 complete [ 2279.393161] LustreError: 75323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2279.421477] LustreError: 75323:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2283.535205] LustreError: 74465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787595715 with bad export cookie 135275076756574247 [ 2283.540889] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2283.547909] LustreError: 74465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2284.167077] Lustre: server umount lustre-MDT0001 complete [ 2296.479915] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 2299.953335] Key type lgssc unregistered [ 2300.249714] LNet: 78266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2300.279043] LNetError: 78266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2300.307813] LNet: Removed LNI 192.168.202.159@tcp [ 2301.323452] Key type .llcrypt unregistered [ 2301.330519] Key type ._llcrypt unregistered [ 2330.050022] Key type ._llcrypt registered [ 2330.052074] Key type .llcrypt registered [ 2330.218235] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2331.465159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2331.573282] alg: No test for adler32 (adler32-zlib) [ 2332.676294] Lustre: Lustre: Build Version: 2.17.57_80_g3abb7b2 [ 2332.894738] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 2334.568188] Key type lgssc registered [ 2336.015889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2344.475859] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2344.493247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2346.027463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2351.988215] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2356.206785] LustreError: 79680:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2356.250353] LustreError: 79680:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2361.326706] LustreError: 79681:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2364.589481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2365.214641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2370.517701] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2380.027580] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2386.881809] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2395.349599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2396.083653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2404.104147] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2413.760584] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2423.810687] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2424.796448] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2425.894243] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2427.980597] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2427.990254] Lustre: Skipped 1 previous similar message [ 2432.076566] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2432.083264] Lustre: Skipped 3 previous similar messages [ 2440.588386] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2440.592637] Lustre: Skipped 8 previous similar messages [ 2456.690930] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2456.696848] Lustre: Skipped 17 previous similar messages [ 2489.243584] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2489.247202] Lustre: Skipped 32 previous similar messages [ 2527.748907] Lustre: 87786:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2531.360987] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 14:26:01 (1787595961) [ 2533.135778] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2535.206292] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 14:26:05 (1787595965) [ 2542.561980] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2542.573034] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2542.611122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2544.613434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2544.617064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2544.626425] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2546.035208] Lustre: server umount lustre-OST0000 complete [ 2554.337867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2554.343115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2554.364891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2554.386078] Lustre: Skipped 1 previous similar message [ 2556.746584] Lustre: server umount lustre-MDT0000 complete [ 2559.973056] LustreError: 80411:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2559.985149] LustreError: 80411:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2560.566967] LustreError: 79660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787595992 with bad export cookie 1629293461223690022 [ 2560.568584] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2560.578101] LustreError: 79660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2561.099668] Lustre: server umount lustre-MDT0001 complete [ 2573.567978] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 2576.395281] Key type lgssc unregistered [ 2576.661314] LNet: 89037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2576.668856] LNetError: 89037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2576.683861] LNet: Removed LNI 192.168.202.159@tcp [ 2577.589955] Key type .llcrypt unregistered [ 2577.592909] Key type ._llcrypt unregistered [ 2582.179249] Key type ._llcrypt registered [ 2582.181215] Key type .llcrypt registered [ 2582.323110] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2586.800309] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules [ 2587.788421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2587.928067] alg: No test for adler32 (adler32-zlib) [ 2589.055836] Lustre: Lustre: Build Version: 2.17.57_80_g3abb7b2 [ 2589.267055] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 2590.928664] Key type lgssc registered [ 2591.980806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2592.945493] LustreError: 90239:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2611.393937] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2614.966906] LustreError: 90239:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2619.626508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2619.640182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2621.292882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2625.825781] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2631.676281] LustreError: 91263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2631.707248] LustreError: 91263:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2636.487917] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 14:27:46 (1787596066) [ 2636.790243] LustreError: 91264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2641.563044] LustreError: 91295:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2641.567990] LustreError: 91295:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2641.584649] LustreError: 91295:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2641.926777] Lustre: server umount lustre-MDT0000 complete [ 2659.754538] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 2662.805352] Key type lgssc unregistered [ 2663.063159] LNet: 92791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2663.073560] LNetError: 92791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2663.092605] LNet: Removed LNI 192.168.202.159@tcp [ 2664.615829] Key type .llcrypt unregistered [ 2664.619562] Key type ._llcrypt unregistered [ 2685.703970] Key type ._llcrypt registered [ 2685.707212] Key type .llcrypt registered [ 2685.889117] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2687.232223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2687.369028] alg: No test for adler32 (adler32-zlib) [ 2688.732374] Lustre: Lustre: Build Version: 2.17.57_80_g3abb7b2 [ 2689.077400] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 2690.728229] Key type lgssc registered [ 2692.024108] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2703.931464] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2703.954855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2705.407472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2712.407805] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2715.619037] LustreError: 94010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2715.643877] LustreError: 94010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2720.739180] LustreError: 94011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2722.996861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2723.549119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2727.789182] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2737.565915] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2744.048980] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2751.953435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2752.510124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2760.302770] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2767.903410] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2831.818701] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 14:31:02 (1787596262) [ 2833.688250] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2835.720747] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 14:31:06 (1787596266) [ 2841.069952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2841.081678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2841.104383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2841.586254] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2841.601174] Lustre: Skipped 1 previous similar message [ 2841.608236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2841.614177] Lustre: Skipped 1 previous similar message [ 2846.692730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2847.572086] Lustre: server umount lustre-MDT0000 complete [ 2851.815514] LustreError: 94006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2851.833471] LustreError: 94006:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2851.840816] LustreError: 93992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787596283 with bad export cookie 8228830803845448440 [ 2851.851829] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2852.208751] Lustre: server umount lustre-MDT0001 complete [ 2867.090216] Lustre: server umount lustre-OST0000 complete [ 2881.134842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2881.836247] LustreError: 98338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2882.015056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2887.143059] LustreError: 98338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2887.174664] LustreError: 98338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2887.769515] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2896.361557] LustreError: 98338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2896.415538] LustreError: 98338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2900.567575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2901.520110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2907.181813] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2916.336325] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2922.405479] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2936.297482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2936.532741] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2936.538299] LustreError: 98328:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2936.571672] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2936.580294] LustreError: 100423:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2936.602401] LustreError: 100423:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2936.607052] LustreError: 100423:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2936.723482] Lustre: server umount lustre-OST0000 complete [ 2936.726556] LustreError: 100423:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2945.485940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2945.646703] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2945.653716] LustreError: 98329:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2945.692648] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2945.712416] LustreError: 100890:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2945.722325] LustreError: 100890:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2945.728954] LustreError: 100890:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2945.840610] Lustre: server umount lustre-OST0000 complete [ 2945.869438] LustreError: 100890:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2957.181122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2957.573147] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2957.580971] Lustre: Found index 0 for lustre-OST0000, updating log [ 2957.586214] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2957.702802] Lustre: lustre-OST0000: new disk, initializing [ 2957.715779] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2957.723888] Lustre: 101426:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2958.206153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2965.431281] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2969.332544] Lustre: Failing over lustre-OST0000 [ 2969.446632] Lustre: server umount lustre-OST0000 complete [ 2974.025545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2974.048993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2974.057944] Lustre: Skipped 3 previous similar messages [ 2979.299050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2979.304093] Lustre: Skipped 1 previous similar message [ 2980.668792] Lustre: server umount lustre-MDT0000 complete [ 2984.421556] LustreError: 98334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2984.442272] LustreError: 98334:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2985.706347] LustreError: 100443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787596417 with bad export cookie 8228830803845451184 [ 2985.715698] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2985.721503] LustreError: 100443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2986.236811] Lustre: server umount lustre-MDT0001 complete [ 2997.773167] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 14:33:47 (1787596427) [ 3019.993676] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3032.973773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3033.460625] LustreError: 103801:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3033.615615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3038.420557] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3047.596993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3053.278522] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3056.264452] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3063.615525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3070.896653] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3080.539902] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3080.636709] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3080.752296] Lustre: lustre-OST0001: new disk, initializing [ 3080.758601] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3080.765903] Lustre: 106119:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3081.090865] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3081.098101] Lustre: Skipped 2 previous similar messages [ 3088.796568] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3091.104687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3091.121980] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3091.275556] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3097.205356] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3102.521580] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3109.857634] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3109.877525] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3109.906658] Lustre: Skipped 1 previous similar message [ 3109.917436] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3114.173322] Lustre: server umount lustre-MDT0001 complete [ 3117.093405] LustreError: 103796:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3117.149489] LustreError: 103796:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3126.770129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3126.877676] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3126.880539] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3126.973280] Lustre: 103789:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3128.590440] LustreError: 107661:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3128.599407] LustreError: 107661:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3128.612391] LustreError: 103793:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3128.762973] LustreError: 107671:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.159@tcp: cfg command failed: rc = -17 [ 3128.774100] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3128.774100] [ 3133.438036] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3134.781658] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3148.637055] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 14:36:18 (1787596578) [ 3171.656848] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3193.177650] Lustre: 109739:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3217.095800] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3221.281609] Lustre: 110873:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3311.610633] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3311.625540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3311.640600] Lustre: Skipped 2 previous similar messages [ 3311.646023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3311.650561] Lustre: Skipped 4 previous similar messages [ 3313.650712] Lustre: server umount lustre-MDT0000 complete [ 3317.730518] LustreError: 103798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3317.775990] LustreError: 103798:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 3317.848188] LustreError: 103783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787596749 with bad export cookie 8228830803845452178 [ 3317.851583] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3317.865487] LustreError: 103783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3318.348280] Lustre: server umount lustre-MDT0001 complete [ 3332.275287] Lustre: server umount lustre-OST0000 complete [ 3346.414155] Lustre: server umount lustre-OST0001 complete [ 3353.027243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3365.962442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3377.201196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3388.374639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3410.511557] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3422.100744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3422.112657] Lustre: lustre-MDT0000: reset Object Index mappings [ 3422.575560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3422.579943] Lustre: Skipped 1 previous similar message [ 3427.250945] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3437.028243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3437.061967] Lustre: lustre-MDT0001: reset Object Index mappings [ 3442.354241] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3445.901279] Lustre: 115974:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3453.276823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3461.327920] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3462.327173] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3467.462915] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3470.536159] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3476.473256] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 3476.478813] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 3477.227432] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3485.833519] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3491.092340] Lustre: 117846:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3547.655155] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 14:42:58 (1787596978) [ 3573.755548] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3573.766437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3573.794310] Lustre: Skipped 5 previous similar messages [ 3573.842755] Lustre: Skipped 7 previous similar messages [ 3580.149515] Lustre: server umount lustre-MDT0000 complete [ 3583.478553] LustreError: 114807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787597015 with bad export cookie 8228830803845499050 [ 3583.483611] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3583.489566] LustreError: 114807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3583.969169] LustreError: 114822:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3583.987752] LustreError: 114822:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 3590.015272] Lustre: server umount lustre-MDT0001 complete [ 3600.632325] Lustre: server umount lustre-OST0000 complete [ 3604.295271] LustreError: 118808:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0001_UUID: namespace resource [0x2c0000400:0x6a:0x0].0x0 (ffff8e1c925fd800) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3610.645599] Lustre: server umount lustre-OST0001 complete [ 3618.404639] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 3634.412624] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3674.909183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3675.183550] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3675.240356] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3675.323254] Lustre: lustre-MDT0000: new disk, initializing [ 3675.425699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3675.440741] Lustre: Skipped 3 previous similar messages [ 3675.466157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3680.542895] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3687.908143] Lustre: server umount lustre-MDT0000 complete [ 3698.712929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3699.293683] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3699.309891] Lustre: 122137:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3699.334075] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3705.448670] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3710.506992] Lustre: server umount lustre-MDT0000 complete [ 3719.998322] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 14:45:50 (1787597150) [ 3722.066290] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3724.596762] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 14:45:54 (1787597154) [ 3744.370822] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3755.710062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3761.013940] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3770.213585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3770.458747] Lustre: 124190:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3770.550568] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3770.560286] Lustre: Skipped 1 previous similar message [ 3770.663867] Lustre: lustre-MDT0001: new disk, initializing [ 3770.832073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3770.850403] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3776.633893] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3782.573556] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3790.922573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3791.311776] Lustre: lustre-OST0000: new disk, initializing [ 3791.320624] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3791.328187] Lustre: 125768:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3792.880846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3792.909721] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3793.043645] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3798.271629] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3809.527837] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3809.737247] Lustre: lustre-OST0001: new disk, initializing [ 3809.741798] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3809.750484] Lustre: 126638:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3811.606384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3811.619471] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3811.802327] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3817.307751] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3828.481714] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3833.239728] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4270.034613] Lustre: 133063:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4270.051604] Lustre: 133063:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4270.063934] Lustre: 133063:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4270.078083] Lustre: 133063:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4270.089739] Lustre: 133063:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4270.095441] Lustre: 133063:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4543.800967] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 14:59:33 (1787597973) [ 4559.334139] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4559.346105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4559.360558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4559.364022] Lustre: Skipped 4 previous similar messages [ 4559.377142] Lustre: Skipped 11 previous similar messages [ 4565.746900] Lustre: server umount lustre-MDT0000 complete [ 4569.575158] LustreError: 124202:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4569.586020] LustreError: 124202:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4569.951618] LustreError: 125767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787598001 with bad export cookie 8228830803845568392 [ 4569.960923] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4569.965503] LustreError: 125767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4574.690390] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4574.702517] Lustre: Skipped 2 previous similar messages [ 4574.712688] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4574.716397] Lustre: Skipped 4 previous similar messages [ 4576.612219] Lustre: server umount lustre-MDT0001 complete [ 4587.350697] Lustre: server umount lustre-OST0000 complete [ 4598.094043] Lustre: server umount lustre-OST0001 complete [ 4605.769704] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 4618.232158] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4657.268065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4657.559380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4657.602629] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4657.656623] Lustre: lustre-MDT0000: new disk, initializing [ 4657.728978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4657.734254] Lustre: Skipped 5 previous similar messages [ 4657.746347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4662.669405] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4673.660549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4673.834231] Lustre: 136123:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4673.880821] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4673.883780] Lustre: Skipped 1 previous similar message [ 4673.963707] Lustre: lustre-MDT0001: new disk, initializing [ 4674.095144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4674.106464] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4678.816315] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4689.678232] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4696.309921] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4704.300040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4704.744192] Lustre: lustre-OST0000: new disk, initializing [ 4704.747896] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4704.754657] Lustre: 138089:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4704.840816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4704.848595] Lustre: Skipped 1 previous similar message [ 4706.799207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4706.823740] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4706.990229] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4711.942207] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4722.902971] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4733.869456] Lustre: lustre-MDD0000: changelog on [ 4736.873632] Lustre: lustre-MDD0001: changelog on [ 6759.667671] Lustre: lustre-MDD0001: changelog off [ 6762.070820] Lustre: 136128:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6763.186565] Lustre: lustre-MDD0000: changelog off [ 6770.145441] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6770.148410] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6770.152822] Lustre: Skipped 1 previous similar message [ 6770.155801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6770.168774] Lustre: Skipped 1 previous similar message [ 6770.664194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6770.675443] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6775.777869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6775.790228] Lustre: Skipped 2 previous similar messages [ 6776.049324] Lustre: server umount lustre-OST0000 complete [ 6780.901641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6780.908792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6780.921040] Lustre: Skipped 1 previous similar message [ 6780.932069] Lustre: Skipped 1 previous similar message [ 6791.138483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6791.147273] Lustre: Skipped 3 previous similar messages [ 6794.209010] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6794.584280] Lustre: server umount lustre-MDT0000 complete [ 6796.263761] LustreError: 136133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6796.303525] LustreError: 136133:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6799.155960] LustreError: 136115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787600231 with bad export cookie 8228830803846341983 [ 6799.163074] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6799.180935] LustreError: 136115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6799.641354] Lustre: server umount lustre-MDT0001 complete [ 6815.949550] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 6819.869101] Key type lgssc unregistered [ 6820.273321] LNet: 147274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6820.279722] LNetError: 147274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6820.293668] LNet: Removed LNI 192.168.202.159@tcp [ 6821.486439] Key type .llcrypt unregistered [ 6821.489742] Key type ._llcrypt unregistered [ 6842.897551] Key type ._llcrypt registered [ 6842.899687] Key type .llcrypt registered [ 6843.037645] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 6860.859600] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 6862.386257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6862.433225] alg: No test for adler32 (adler32-zlib) [ 6863.492683] Lustre: Lustre: Build Version: 2.17.57_80_g3abb7b2 [ 6863.752796] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 6865.472321] Key type lgssc registered [ 6866.717672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6907.568818] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6907.623508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6909.014608] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6909.102811] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6909.222323] Lustre: lustre-MDT0000: new disk, initializing [ 6909.377354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6909.402837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6914.494381] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6925.070707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6925.302588] Lustre: 150476:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6925.399119] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6925.406021] Lustre: Skipped 1 previous similar message [ 6925.524117] Lustre: lustre-MDT0001: new disk, initializing [ 6925.626573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6925.651491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6925.669839] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6930.796461] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6941.894778] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6948.618250] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6956.740820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6957.003574] Lustre: lustre-OST0000: new disk, initializing [ 6957.007248] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6957.013491] Lustre: 152443:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6957.110400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6963.869656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6963.886192] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6964.091677] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6965.371581] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6977.244277] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6985.391469] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_module obdecho/obdecho [ 6987.959916] LustreError: 153449:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6987.963767] LustreError: 153449:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6990.296807] LustreError: 153547:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7002.593425] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7002.624851] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7002.661668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7005.154918] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7005.156541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7008.770609] Lustre: server umount lustre-MDT0000 complete [ 7014.454064] LustreError: 150468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787600446 with bad export cookie 12224495335497688652 [ 7014.460035] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7014.476063] LustreError: 150468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7015.269270] Lustre: server umount lustre-MDT0001 complete [ 7031.531287] Lustre: server umount lustre-OST0000 complete [ 7060.675860] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 7076.831617] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 7121.893072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7122.176900] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7122.213573] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7122.313131] Lustre: lustre-MDT0000: new disk, initializing [ 7122.397126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7122.418284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7128.234222] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7142.966125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7143.092334] Lustre: 157288:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7143.139774] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7143.144762] Lustre: Skipped 1 previous similar message [ 7143.228332] Lustre: lustre-MDT0001: new disk, initializing [ 7143.398292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7143.469135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7143.494391] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7148.761106] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7159.392473] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7166.961418] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7175.421819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7175.679904] Lustre: lustre-OST0000: new disk, initializing [ 7175.684196] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7175.689446] Lustre: 159258:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7175.788172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7181.950728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7181.967717] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7182.113974] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7183.226910] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7195.353344] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7202.596110] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7202.929145] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7208.574811] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7208.774406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7213.025179] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7213.027051] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7213.046284] LustreError: Skipped 1 previous similar message [ 7213.051656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7213.051665] Lustre: Skipped 2 previous similar messages [ 7213.088542] Lustre: Skipped 2 previous similar messages [ 7218.151701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7218.161748] Lustre: Skipped 2 previous similar messages [ 7223.270133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7225.312384] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7225.736974] Lustre: server umount lustre-OST0000 complete [ 7233.510066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7233.531806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7233.543405] Lustre: Skipped 2 previous similar messages [ 7237.674978] Lustre: server umount lustre-MDT0000 complete [ 7238.626766] LustreError: 157298:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7238.651578] LustreError: 157298:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7242.155771] LustreError: 157279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787600674 with bad export cookie 12224495335497689611 [ 7242.158262] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7242.171021] LustreError: 157279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7242.676448] Lustre: server umount lustre-MDT0001 complete [ 7248.044856] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 15:44:37 (1787600677) [ 7250.027045] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7252.293317] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 15:44:42 (1787600682) [ 7260.937048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7261.300220] LustreError: 161673:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7261.364100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7266.330424] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7266.788339] LustreError: 161674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7271.913403] LustreError: 161673:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7276.686761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7282.493621] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7291.729930] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7298.226717] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7305.479756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7305.795605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7305.802599] Lustre: Skipped 1 previous similar message [ 7312.644261] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7321.510207] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7331.297642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7331.310397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7331.318768] Lustre: Skipped 3 previous similar messages [ 7331.336407] Lustre: Skipped 2 previous similar messages [ 7334.237395] Lustre: server umount lustre-MDT0000 complete [ 7336.421059] LustreError: 161670:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7336.442971] LustreError: 161670:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7338.565899] LustreError: 161655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787600770 with bad export cookie 12224495335497690493 [ 7338.569380] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7338.575349] LustreError: 161655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7341.538953] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7345.491628] Lustre: server umount lustre-MDT0001 complete [ 7354.927066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7373.979031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7374.176436] LustreError: 165517:0:(llog.c:1655:llog_backup()) MGC192.168.202.159@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7374.184437] Lustre: 165517:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.159@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7382.849112] LustreError: 165522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7382.887868] LustreError: 165522:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7398.884684] LustreError: 165522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7398.910534] LustreError: 165522:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 7414.922633] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.59@tcp (not set up) [ 7414.927298] Lustre: Skipped 4 previous similar messages [ 7433.701855] LustreError: 165524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7433.738598] LustreError: 165524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 25 previous similar messages [ 7435.041447] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7435.050657] Lustre: 165517:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.159@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7475.745765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7480.820828] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7482.993704] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7493.762711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7499.236683] LustreError: 165559:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7499.250388] LustreError: 165559:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 46 previous similar messages [ 7504.367703] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7508.485365] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xa9a621bd4ec286d6 [ 7508.502413] LustreError: 166275:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.202.159@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (0000000001cde637) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 7508.522462] Lustre: MGC192.168.202.159@tcp: Connection restored to 0@lo (at 0@lo) [ 7514.121614] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7515.265991] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7524.900051] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7527.838793] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7537.555170] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7540.430568] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7565.282866] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7565.286129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7565.313827] Lustre: Skipped 1 previous similar message [ 7565.333791] Lustre: Skipped 2 previous similar messages [ 7569.740524] Lustre: server umount lustre-MDT0000 complete [ 7574.043856] LustreError: 161698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787601006 with bad export cookie 12224495335497696982 [ 7574.045741] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7574.083554] LustreError: 161698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7574.552530] Lustre: server umount lustre-MDT0001 complete [ 7588.334618] Lustre: server umount lustre-OST0000 complete [ 7613.149368] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 7625.208766] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 7671.701654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7672.009915] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7672.039907] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7672.110884] Lustre: lustre-MDT0000: new disk, initializing [ 7672.166562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7672.176382] Lustre: Skipped 1 previous similar message [ 7672.193564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7677.835677] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7690.244415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7690.425588] Lustre: 170631:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7690.675982] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7695.566208] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7706.811724] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7714.103364] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7721.975377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7722.243433] Lustre: lustre-OST0000: new disk, initializing [ 7722.246174] Lustre: Skipped 1 previous similar message [ 7722.256986] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7722.268463] Lustre: Skipped 2 previous similar messages [ 7722.281484] Lustre: 172598:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7727.690463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7727.707099] Lustre: Skipped 1 previous similar message [ 7727.726249] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7727.870908] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7729.706136] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7740.540666] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7745.498956] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7745.728488] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7750.506905] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7750.700192] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7753.701997] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7753.710626] Lustre: Skipped 1 previous similar message [ 7753.718048] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7753.720155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7753.753110] Lustre: Skipped 1 previous similar message [ 7759.256668] Lustre: server umount lustre-OST0000 complete [ 7778.276219] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7778.483662] Lustre: server umount lustre-MDT0000 complete [ 7779.299401] LustreError: 170637:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7779.317925] LustreError: 170637:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 7783.097384] LustreError: 170622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787601215 with bad export cookie 12224495335497713782 [ 7783.103077] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7783.118341] LustreError: 170622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7783.629728] Lustre: server umount lustre-MDT0001 complete [ 7794.749266] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 15:53:44 (1787601224) [ 7802.280411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7807.108995] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7815.941542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7821.001641] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7829.385297] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7835.523893] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7843.039769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7850.414101] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7858.654705] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7873.480888] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 15:55:03 (1787601303) [ 7878.630513] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7878.635321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7878.649727] Lustre: Skipped 3 previous similar messages [ 7884.343100] Lustre: server umount lustre-OST0000 complete [ 7895.720800] Lustre: server umount lustre-MDT0000 complete [ 7900.141276] LustreError: 174994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787601332 with bad export cookie 12224495335497714664 [ 7900.150308] LustreError: 174994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7900.151629] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7900.741538] Lustre: server umount lustre-MDT0001 complete [ 7915.600261] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 7918.460608] Key type lgssc unregistered [ 7918.708611] LNet: 178746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7918.713036] LNetError: 178746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7918.731818] LNet: Removed LNI 192.168.202.159@tcp [ 7919.757454] Key type .llcrypt unregistered [ 7919.759317] Key type ._llcrypt unregistered [ 7942.089165] Key type ._llcrypt registered [ 7942.091194] Key type .llcrypt registered [ 7942.170765] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 7943.197646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7943.476418] alg: No test for adler32 (adler32-zlib) [ 7944.441459] Lustre: Lustre: Build Version: 2.17.57_80_g3abb7b2 [ 7944.661225] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 7946.360199] Key type lgssc registered [ 7947.411695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7955.556158] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7955.590124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7971.169781] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7976.290484] LustreError: 179991:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.159@tcp: failed processing log, type 4: rc = -110 [ 7981.410361] LustreError: 179991:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7981.421920] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7981.436012] LustreError: MGC192.168.202.159@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7981.449354] LustreError: 179991:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7981.459601] LustreError: 179991:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 7981.471270] LustreError: 179991:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7981.563529] Lustre: server umount lustre-OST0000 complete [ 7981.568528] LustreError: 179991:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7992.319693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7994.144659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7999.422798] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8004.603995] LustreError: 180491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8004.621677] LustreError: 180491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8009.216958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8009.637679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8015.585946] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8025.470661] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8033.055828] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8040.526557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8040.671324] LustreError: 182306:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8040.677593] Lustre: 182306:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.159@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8040.762165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8047.620185] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8057.015690] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8060.901517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8060.912265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8060.950848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8064.499353] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8064.500514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8064.531111] Lustre: Skipped 2 previous similar messages [ 8066.123034] Lustre: server umount lustre-MDT0000 complete [ 8069.616184] LustreError: 180485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8069.640173] LustreError: 180472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787601501 with bad export cookie 8965652016666629112 [ 8069.649512] LustreError: 180485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8069.654525] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8069.663061] LustreError: 180472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8070.033279] Lustre: server umount lustre-MDT0001 complete [ 8085.549464] Lustre: server umount lustre-OST0000 complete [ 8096.105496] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 8107.166923] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 8152.240523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8152.578946] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8152.600980] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8152.773229] Lustre: lustre-MDT0000: new disk, initializing [ 8152.870033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8152.895299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8158.160444] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8170.981939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8171.153981] Lustre: 185839:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8171.264545] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8171.268582] Lustre: Skipped 1 previous similar message [ 8171.354756] Lustre: lustre-MDT0001: new disk, initializing [ 8171.510443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8171.587823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8171.617387] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8176.015935] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8185.845880] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8193.267195] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8199.607842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8199.989898] Lustre: lustre-OST0000: new disk, initializing [ 8199.996738] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8200.009625] Lustre: 187806:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8200.117440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8207.434864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8207.447570] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8207.541478] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8207.579808] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8216.806305] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8221.858688] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8221.977851] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8226.268654] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8226.453856] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8233.445684] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8233.451172] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8233.464338] Lustre: Skipped 1 previous similar message [ 8233.471409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8233.491692] Lustre: Skipped 1 previous similar message [ 8234.577994] Lustre: server umount lustre-OST0000 complete [ 8238.575805] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8238.588482] Lustre: Skipped 1 previous similar message [ 8238.594158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8243.169400] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8243.181184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8243.200447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8244.833288] Lustre: server umount lustre-MDT0000 complete [ 8248.805195] LustreError: 185849:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8248.822951] LustreError: 185849:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8249.140526] LustreError: 187817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787601681 with bad export cookie 8965652016666629826 [ 8249.159578] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8249.549971] Lustre: server umount lustre-MDT0001 complete [ 8260.890481] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 16:01:31 (1787601691) [ 8278.865522] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 8281.348345] Key type lgssc unregistered [ 8281.580281] LNet: 190765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8281.584655] LNetError: 190765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8281.601700] LNet: Removed LNI 192.168.202.159@tcp [ 8282.363443] Key type .llcrypt unregistered [ 8282.372243] Key type ._llcrypt unregistered [ 8304.386866] Key type ._llcrypt registered [ 8304.390911] Key type .llcrypt registered [ 8304.514399] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 8305.398070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8305.479489] alg: No test for adler32 (adler32-zlib) [ 8306.562114] Lustre: Lustre: Build Version: 2.17.57_80_g3abb7b2 [ 8306.903487] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 8308.568198] Key type lgssc registered [ 8309.799445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8317.219637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8317.248376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8318.719108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8323.298228] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8328.163567] LustreError: 191982:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8328.191638] LustreError: 191982:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8332.907758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8333.283611] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8333.463764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8337.988229] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8347.026763] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8352.736741] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8358.815602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8358.949289] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8358.951279] LustreError: 193846:0:(llog.c:1696:llog_backup()) MGC192.168.202.159@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8358.956477] Lustre: 193846:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.159@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8359.063435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8364.743680] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8371.804280] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8374.241738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8374.249364] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8374.263849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8378.402845] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8378.405595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8378.422731] Lustre: Skipped 1 previous similar message [ 8378.438889] Lustre: Skipped 2 previous similar messages [ 8380.389766] Lustre: server umount lustre-MDT0000 complete [ 8383.463327] LustreError: 191982:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8383.487264] LustreError: 191982:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8383.678547] LustreError: 191962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787601815 with bad export cookie 8743102152450525882 [ 8383.686851] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8383.692382] LustreError: 191962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8384.608289] LustreError: 192713:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000088ed400d x1874436280052352/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8384.766689] Lustre: server umount lustre-MDT0001 complete [ 8398.653753] Lustre: server umount lustre-OST0000 complete [ 8409.448710] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 8419.639938] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 8455.469861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8455.730462] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8455.755084] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8455.840733] Lustre: lustre-MDT0000: new disk, initializing [ 8455.910298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8455.939108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8460.638933] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8472.394513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8472.568512] Lustre: 197369:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8472.640363] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8472.644434] Lustre: Skipped 1 previous similar message [ 8472.646677] LustreError: 198154:0:(llog.c:1696:llog_backup()) MGC192.168.202.159@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8472.653794] LustreError: 198154:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8472.659576] Lustre: 198154:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.159@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8472.677977] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8472.682143] Lustre: Skipped 1 previous similar message [ 8472.767520] Lustre: lustre-MDT0001: new disk, initializing [ 8472.857119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8472.896940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8472.907281] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8476.933500] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8486.783267] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8493.900176] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8503.517579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8503.870361] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8503.884165] Lustre: Skipped 1 previous similar message [ 8503.897719] LustreError: 199316:0:(llog.c:1696:llog_backup()) MGC192.168.202.159@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8503.915208] LustreError: 199316:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8503.928064] Lustre: 199316:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.159@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8503.940497] Lustre: 199316:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8503.980593] Lustre: lustre-OST0000: new disk, initializing [ 8503.984716] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8503.991814] Lustre: 199340:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8504.161139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8509.541261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8509.556467] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8509.709171] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8512.197264] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8523.024948] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8528.047703] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8528.178643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8532.777422] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8532.978910] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8535.522138] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8535.522954] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8535.526687] LustreError: Skipped 1 previous similar message [ 8535.530380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8535.552273] Lustre: Skipped 1 previous similar message [ 8535.576787] Lustre: Skipped 1 previous similar message [ 8540.648920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8540.652108] Lustre: Skipped 1 previous similar message [ 8549.344208] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8549.575578] Lustre: server umount lustre-OST0000 complete [ 8554.990188] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8555.003563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8555.019789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8555.031694] Lustre: Skipped 2 previous similar messages [ 8567.264197] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8567.500495] Lustre: server umount lustre-MDT0000 complete [ 8570.862438] LustreError: 197364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787602002 with bad export cookie 8743102152450526596 [ 8570.864257] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8570.875291] LustreError: 197364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8571.177691] Lustre: server umount lustre-MDT0001 complete [ 8580.529695] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 16:06:51 (1787602011) [ 8600.183158] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 8612.104935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8612.564287] LustreError: 202473:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8612.620514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8617.953543] LustreError: 202474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8618.422636] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8622.053862] LustreError: 202473:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8628.344141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8633.782661] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8636.725994] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8642.925302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8649.570950] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8659.083380] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8659.285243] Lustre: lustre-OST0001: new disk, initializing [ 8659.289341] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8659.295308] Lustre: 204781:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8659.423952] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8659.432102] Lustre: Skipped 2 previous similar messages [ 8665.679836] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8666.780846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8666.787819] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8666.905304] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8677.536484] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8688.426558] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8691.328983] Lustre: DEBUG MARKER: ADD OST3 [ 8696.792297] loop: module loaded [ 8696.853215] loop0: detected capacity change from 0 to 204800000 [ 8697.038922] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8697.049477] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8697.073650] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8703.661173] loop0: detected capacity change from 0 to 204800000 [ 8704.439717] Lustre: DEBUG MARKER: STOP OST3 [ 8709.940603] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8710.085961] Lustre: lustre-OST0003: new disk, initializing [ 8710.088865] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8710.095339] Lustre: 206799:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8711.979405] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8712.203245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8712.221051] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8712.267676] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8714.016349] LustreError: 206955:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8714.127555] Lustre: Failing over lustre-OST0003 [ 8714.391067] Lustre: server umount lustre-OST0003 complete [ 8716.980783] LustreError: 205213:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8717.005946] LustreError: 205213:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8717.284797] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8717.295689] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8717.312195] Lustre: Skipped 1 previous similar message [ 8717.467104] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8725.658319] Lustre: DEBUG MARKER: START OST3 again [ 8732.060497] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8732.306296] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8732.313583] Lustre: Skipped 1 previous similar message [ 8732.324246] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8733.676057] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8734.012205] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8734.012524] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8738.174957] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8760.650333] Lustre: Failing over lustre-OST0003 [ 8760.792307] Lustre: server umount lustre-OST0003 complete [ 8761.325883] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8761.346197] LustreError: 203958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8761.354638] Lustre: Skipped 1 previous similar message [ 8761.371390] LustreError: 203958:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 8781.798900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8781.800376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8781.826610] Lustre: Skipped 3 previous similar messages [ 8781.836725] Lustre: Skipped 9 previous similar messages [ 8784.159254] Lustre: server umount lustre-MDT0000 complete [ 8787.321875] LustreError: 202453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787602219 with bad export cookie 8743102152450527485 [ 8787.324026] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8787.341450] LustreError: 202453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8793.936544] Lustre: server umount lustre-MDT0001 complete [ 8797.562788] Lustre: server umount lustre-OST0000 complete [ 8800.927691] Lustre: server umount lustre-OST0001 complete [ 8808.090805] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 8816.591349] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 8856.981368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8857.246467] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8857.286355] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8857.399859] Lustre: lustre-MDT0000: new disk, initializing [ 8857.487028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8862.491601] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8874.357628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8874.501924] Lustre: 212241:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8874.533903] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8874.537893] Lustre: Skipped 1 previous similar message [ 8874.614843] Lustre: lustre-MDT0001: new disk, initializing [ 8874.738337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8874.759746] Lustre: Skipped 1 previous similar message [ 8874.793599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8874.811392] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8879.794049] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8889.757959] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8895.637944] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8901.536227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8901.745299] Lustre: 214211:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8903.152361] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8903.274087] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8907.297544] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8917.819985] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8923.049057] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8923.392328] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8929.278970] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8929.480238] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8933.864359] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8933.868823] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8933.876229] Lustre: Skipped 2 previous similar messages [ 8933.883292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8933.888325] Lustre: Skipped 2 previous similar messages [ 8937.748432] Lustre: server umount lustre-OST0000 complete [ 8948.743282] Lustre: server umount lustre-MDT0000 complete [ 8949.234255] LustreError: 214937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8949.282144] LustreError: 214937:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 8954.013848] LustreError: 213036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787602386 with bad export cookie 8743102152450529774 [ 8954.035110] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8954.037992] LustreError: 213036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8954.444630] Lustre: server umount lustre-MDT0001 complete [ 8959.447225] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 16:13:09 (1787602389) [ 8979.423336] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 8996.824497] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 9034.247751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9034.547804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9034.572976] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9034.580179] Lustre: Skipped 1 previous similar message [ 9034.693049] Lustre: lustre-MDT0000: new disk, initializing [ 9034.694974] Lustre: Skipped 1 previous similar message [ 9034.772073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9034.778238] Lustre: Skipped 1 previous similar message [ 9039.372407] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9051.697716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9051.855960] Lustre: 218844:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9052.165232] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9056.671057] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9066.262209] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9072.017369] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9079.370670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9079.670281] Lustre: 220812:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9081.707725] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9087.384822] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9097.427835] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9102.605514] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9102.873230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9107.033676] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9107.214754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9112.545526] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9112.549154] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9112.552135] LustreError: Skipped 2 previous similar messages [ 9112.552666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9112.552670] Lustre: Skipped 2 previous similar messages [ 9112.596141] Lustre: Skipped 4 previous similar messages [ 9115.356722] Lustre: server umount lustre-OST0000 complete [ 9125.334410] Lustre: server umount lustre-MDT0000 complete [ 9127.913719] LustreError: 218852:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9127.966054] LustreError: 218852:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9129.083222] LustreError: 218836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787602561 with bad export cookie 8743102152450530663 [ 9129.101616] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9129.104674] LustreError: 218836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9129.579991] Lustre: server umount lustre-MDT0001 complete [ 9138.297324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9138.744034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9138.747372] Lustre: Skipped 4 previous similar messages [ 9143.904690] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9154.358871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9160.040859] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9167.702805] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9173.437938] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9180.455803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9188.117693] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9196.970328] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9209.314471] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9209.342891] LustreError: Skipped 1 previous similar message [ 9213.586310] Lustre: server umount lustre-OST0000 complete [ 9223.777645] Lustre: server umount lustre-MDT0000 complete [ 9228.039936] LustreError: 223009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787602660 with bad export cookie 8743102152450531545 [ 9228.049363] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9228.053132] LustreError: 223009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9228.642842] Lustre: server umount lustre-MDT0001 complete [ 9339.491573] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 9342.626824] Key type lgssc unregistered [ 9343.012126] LNet: 226701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9343.041174] LNetError: 226701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9343.061994] LNet: Removed LNI 192.168.202.159@tcp [ 9344.088209] Key type .llcrypt unregistered [ 9344.090390] Key type ._llcrypt unregistered [ 9374.329931] Key type ._llcrypt registered [ 9374.333949] Key type .llcrypt registered [ 9374.512680] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 9393.784581] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 9394.429690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9394.455892] alg: No test for adler32 (adler32-zlib) [ 9395.874281] Lustre: Lustre: Build Version: 2.17.57_80_g3abb7b2 [ 9396.177973] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 9397.856389] Key type lgssc registered [ 9399.273697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9436.070807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9436.118703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9437.501332] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9437.530934] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9437.639821] Lustre: lustre-MDT0000: new disk, initializing [ 9437.752519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9437.805893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9442.503423] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9453.045519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9453.237342] Lustre: 230057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9453.348878] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9453.356258] Lustre: Skipped 1 previous similar message [ 9453.436831] Lustre: lustre-MDT0001: new disk, initializing [ 9453.558558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9453.618653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9453.639430] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9457.705340] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9467.794711] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9473.561551] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9480.511107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9480.900765] Lustre: lustre-OST0000: new disk, initializing [ 9480.911739] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9480.921542] Lustre: 232022:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9481.029545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9487.421383] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9491.068116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9491.080022] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9491.307418] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9498.234579] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9502.920527] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9503.086813] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9507.752320] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9507.951975] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9511.909251] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9511.910595] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9511.924944] LustreError: Skipped 1 previous similar message [ 9511.926728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9511.938584] Lustre: Skipped 1 previous similar message [ 9511.950355] Lustre: Skipped 1 previous similar message [ 9517.038438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9517.056395] Lustre: Skipped 1 previous similar message [ 9522.147804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9522.154739] Lustre: Skipped 1 previous similar message [ 9524.194595] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9524.386643] Lustre: server umount lustre-OST0000 complete [ 9530.338885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9530.345231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9530.387200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9532.387573] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9534.959512] Lustre: server umount lustre-MDT0000 complete [ 9539.505107] LustreError: 230047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787602971 with bad export cookie 4130877715643636635 [ 9539.509277] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9539.513539] LustreError: 230047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9539.737594] Lustre: server umount lustre-MDT0001 complete [ 9562.061150] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 9574.459913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9574.950041] LustreError: 234958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9574.975584] LustreError: 234958:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9575.027177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9579.364664] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9580.514099] LustreError: 234959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9580.522831] LustreError: 234959:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9585.642017] LustreError: 234958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9586.612559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9586.900786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9590.933552] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9593.521250] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9599.536460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9599.836819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9605.866642] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9614.635067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9614.788511] Lustre: lustre-OST0001: new disk, initializing [ 9614.795700] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9614.800530] Lustre: 237265:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9620.584963] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9622.122897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9622.135270] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9622.233264] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9633.709564] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9641.929913] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9657.795342] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 16:24:48 (1787603088) [ 9663.459527] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9663.470525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9663.481274] Lustre: Skipped 5 previous similar messages [ 9665.568731] Lustre: server umount lustre-MDT0000 complete [ 9668.585156] LustreError: 236236:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9668.601670] LustreError: 236236:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9669.056219] LustreError: 236466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787603101 with bad export cookie 4130877715643637517 [ 9669.062344] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9669.066301] LustreError: 236466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9669.831686] Lustre: server umount lustre-MDT0001 complete [ 9683.293777] Lustre: server umount lustre-OST0000 complete [ 9696.072650] Lustre: server umount lustre-OST0001 complete [ 9703.737958] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 9715.515964] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 9755.164277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9755.513428] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9755.551751] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9755.702695] Lustre: lustre-MDT0000: new disk, initializing [ 9755.794349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9755.797087] Lustre: Skipped 1 previous similar message [ 9755.814169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9760.641255] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9772.409339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9772.578467] Lustre: 241214:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9772.640700] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9772.644695] Lustre: Skipped 1 previous similar message [ 9772.748533] Lustre: lustre-MDT0001: new disk, initializing [ 9772.957969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9772.990943] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9778.959597] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9790.859313] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9798.270654] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9805.338707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9805.601685] Lustre: lustre-OST0000: new disk, initializing [ 9805.604710] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9805.609538] Lustre: 243187:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9805.683107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9805.694783] Lustre: Skipped 1 previous similar message [ 9807.027785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9807.054943] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9807.255673] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9813.746598] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9822.910689] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9828.014386] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9828.173620] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9832.449684] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9832.629405] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9838.050025] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9838.055636] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9838.057193] LustreError: Skipped 1 previous similar message [ 9838.058305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9838.080017] Lustre: Skipped 4 previous similar messages [ 9838.098157] Lustre: Skipped 1 previous similar message [ 9840.604425] Lustre: server umount lustre-OST0000 complete [ 9844.706615] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9848.298727] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9848.314159] Lustre: Skipped 1 previous similar message [ 9853.001423] Lustre: server umount lustre-MDT0000 complete [ 9853.411591] LustreError: 241223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9853.446061] LustreError: 241223:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9857.052655] LustreError: 241208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787603289 with bad export cookie 4130877715643639645 [ 9857.061853] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9857.065456] LustreError: 241208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9857.793498] Lustre: server umount lustre-MDT0001 complete [ 9867.466144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9867.915844] LustreError: 245403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9872.272441] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9880.986656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9881.472752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9881.487768] Lustre: Skipped 1 previous similar message [ 9886.084222] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9894.632358] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9901.909335] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9909.767666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9916.994403] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9924.818698] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10069.988778] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10069.997168] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10070.006562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10070.024538] Lustre: Skipped 4 previous similar messages [10084.320285] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10084.495238] Lustre: server umount lustre-OST0000 complete [10095.071138] Lustre: server umount lustre-MDT0000 complete [10095.586400] LustreError: 245403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10095.613410] LustreError: 245403:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10098.697191] LustreError: 245383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787603530 with bad export cookie 4130877715643640534 [10098.709401] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10098.715165] LustreError: 245383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10099.300917] Lustre: server umount lustre-MDT0001 complete [10195.200480] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [10198.465610] Key type lgssc unregistered [10198.710134] LNet: 249015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10198.714915] LNetError: 249015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10198.725648] LNet: Removed LNI 192.168.202.159@tcp [10199.645210] Key type .llcrypt unregistered [10199.649317] Key type ._llcrypt unregistered [10229.562521] Key type ._llcrypt registered [10229.565847] Key type .llcrypt registered [10229.657313] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [10249.101384] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [10251.042703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10251.102482] alg: No test for adler32 (adler32-zlib) [10252.432523] Lustre: Lustre: Build Version: 2.17.57_80_g3abb7b2 [10252.794447] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [10254.624830] Key type lgssc registered [10256.027200] Lustre: Echo OBD driver; http://www.lustre.org/ [10310.434283] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [10322.125381] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10322.149486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10323.515853] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10323.560388] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10323.653738] Lustre: lustre-MDT0000: new disk, initializing [10323.741522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10323.756777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10329.047442] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10341.264822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10341.475749] Lustre: 253091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10341.569344] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10341.575667] Lustre: Skipped 1 previous similar message [10341.695727] Lustre: lustre-MDT0001: new disk, initializing [10341.854617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10341.898206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10341.914697] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10346.901585] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10352.249087] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10360.074121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10360.365669] Lustre: lustre-OST0000: new disk, initializing [10360.369656] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10360.373930] Lustre: 254721:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10360.471727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10367.596867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10367.612521] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10367.823194] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10367.857501] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10379.700430] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10379.862740] Lustre: lustre-OST0001: new disk, initializing [10379.867834] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10379.875776] Lustre: 255592:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10380.007386] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10388.429052] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10388.593078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10388.605816] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10388.701295] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10400.567714] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10408.435786] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10423.780371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10423.785250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10423.814799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10424.808678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10424.840502] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10424.848298] Lustre: Skipped 2 previous similar messages [10428.889222] Lustre: server umount lustre-MDT0000 complete [10432.820397] LustreError: 253082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787603864 with bad export cookie 1757421729986593623 [10432.827596] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10432.832166] LustreError: 253082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10433.168988] Lustre: server umount lustre-MDT0001 complete [10448.821865] Lustre: server umount lustre-OST0000 complete [10451.232098] Lustre: 250594:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787603867/real 1787603867] req@ffff8e1caec43480 x1874438320633344/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787603883 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10451.273641] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10452.832732] Lustre: server umount lustre-OST0001 complete [10467.400303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10477.577381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10486.156606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10496.610628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10520.172744] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [10530.985496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10531.021219] Lustre: lustre-MDT0000: reset Object Index mappings [10531.597219] LustreError: 260605:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10531.614772] LustreError: 260605:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10531.743887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10536.933184] LustreError: 260606:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10537.428949] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10541.044593] LustreError: 260605:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10546.150720] LustreError: 260606:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10547.959281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10547.974938] Lustre: lustre-MDT0001: reset Object Index mappings [10548.472277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10554.532313] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10557.868880] Lustre: 261749:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10566.114321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10566.719992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10574.712655] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10574.894614] LustreError: 262103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10576.951222] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10583.039592] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10588.660456] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10589.559248] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10597.648458] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10601.234612] Lustre: 263619:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10608.312446] Lustre: server umount lustre-MDT0000 complete [10609.129442] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10609.142326] LustreError: 260602:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10609.148602] Lustre: Skipped 2 previous similar messages [10609.177565] LustreError: 260602:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10612.067789] LustreError: 260588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787604044 with bad export cookie 1757421729986600413 [10612.073657] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10612.077719] LustreError: 260588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10612.531280] Lustre: server umount lustre-MDT0001 complete [10626.669723] Lustre: server umount lustre-OST0000 complete [10630.624148] Lustre: 250596:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787604046/real 1787604046] req@ffff8e1cbf9c6680 x1874438320684288/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787604062 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10630.662536] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10630.682765] Lustre: Skipped 1 previous similar message [10631.123499] Lustre: server umount lustre-OST0001 complete [10649.925787] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 16:41:20 (1787604080) [10666.856973] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [10682.072048] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [10716.944401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10717.101848] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10717.149397] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10717.244431] Lustre: lustre-MDT0000: new disk, initializing [10717.287485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10717.291018] Lustre: Skipped 1 previous similar message [10717.299796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10721.572974] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10732.639515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10732.757305] Lustre: 267890:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10732.811073] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10732.815495] Lustre: Skipped 1 previous similar message [10732.885711] Lustre: lustre-MDT0001: new disk, initializing [10733.021175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10733.045423] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10738.484447] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10749.433985] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10755.739254] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10763.655384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10763.922199] Lustre: lustre-OST0000: new disk, initializing [10763.927122] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10763.935290] Lustre: 269861:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10765.528138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10765.555313] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10765.698588] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10770.522277] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10781.996819] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10788.528431] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10788.875313] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10795.946828] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10796.301950] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10801.633852] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10801.636214] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10801.639489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10801.639495] Lustre: Skipped 3 previous similar messages [10801.654494] Lustre: Skipped 1 previous similar message [10804.970042] Lustre: server umount lustre-OST0000 complete [10809.829300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10809.837356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10809.845053] Lustre: Skipped 1 previous similar message [10815.696530] Lustre: server umount lustre-MDT0000 complete [10816.997357] LustreError: 267898:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10817.050433] LustreError: 267898:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10820.523801] LustreError: 267883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787604252 with bad export cookie 1757421729986602506 [10820.533587] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10820.536145] LustreError: 267883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10820.855150] Lustre: server umount lustre-MDT0001 complete [10841.593334] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [10852.886387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10853.365176] LustreError: 272796:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10853.423488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10853.429358] Lustre: Skipped 2 previous similar messages [10858.868849] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10867.372090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10872.434640] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10875.241310] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10881.863500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10889.431592] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10899.576399] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10899.757825] Lustre: lustre-OST0001: new disk, initializing [10899.761582] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10899.768739] Lustre: 275105:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10901.740715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10901.749826] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10901.893416] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10906.420803] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10917.452252] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10921.071677] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10927.356330] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10927.375258] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10934.242055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10934.253871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10934.268179] Lustre: Skipped 2 previous similar messages [10934.285205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10934.292666] Lustre: Skipped 2 previous similar messages [10940.357720] Lustre: server umount lustre-MDT0000 complete [10942.948271] LustreError: 272796:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10942.983451] LustreError: 272796:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10945.950903] LustreError: 272778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787604377 with bad export cookie 1757421729986603388 [10945.957023] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10945.977891] LustreError: 272778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10946.617697] Lustre: server umount lustre-MDT0001 complete [10962.015173] Lustre: server umount lustre-OST0000 complete [10964.768965] Lustre: 250595:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787604380/real 1787604380] req@ffff8e1cb6969880 x1874438320794880/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787604396 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10966.873514] Lustre: server umount lustre-OST0001 complete [10991.113278] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [11006.080168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11006.421641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11006.423808] Lustre: Skipped 3 previous similar messages [11011.495735] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11021.786836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11028.278358] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11031.944128] Lustre: 279301:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11040.706055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11050.499772] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11051.588730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11062.211343] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11067.424800] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11071.122699] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11079.378190] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11083.446731] Lustre: 281183:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11089.438370] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11089.468194] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11101.209542] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 16:48:50 (1787604530) [11118.568726] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11118.579527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11118.592021] Lustre: Skipped 7 previous similar messages [11118.595505] Lustre: Skipped 6 previous similar messages [11123.543278] Lustre: server umount lustre-MDT0000 complete [11123.683539] LustreError: 278157:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11123.715938] LustreError: 278157:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [11128.310301] LustreError: 278143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787604560 with bad export cookie 1757421729986609926 [11128.317964] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11128.343328] LustreError: 278143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11134.898166] Lustre: server umount lustre-MDT0001 complete [11153.376397] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11153.647349] Lustre: server umount lustre-OST0000 complete [11158.136254] Lustre: server umount lustre-OST0001 complete [11166.615386] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [11179.546329] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [11237.048826] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [11249.317405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11249.732221] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11249.794540] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11249.983821] Lustre: lustre-MDT0000: new disk, initializing [11250.151039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11256.183553] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11268.505532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11268.683197] Lustre: 285450:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11268.956888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11268.963099] Lustre: Skipped 4 previous similar messages [11269.028141] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11274.714610] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11287.622456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11288.033935] Lustre: 287086:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11289.773528] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11295.922775] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11308.082961] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11308.350203] Lustre: 287955:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11309.624658] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11309.642988] Lustre: Skipped 1 previous similar message [11309.829374] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11316.099285] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11327.124994] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11331.915118] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11331.923244] Lustre: Skipped 1 previous similar message [11348.631193] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 16:52:58 (1787604778) [11356.137623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11356.139474] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11356.155429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11356.155442] Lustre: Skipped 5 previous similar messages [11356.220090] Lustre: Skipped 5 previous similar messages [11357.756717] Lustre: server umount lustre-MDT0000 complete [11361.291177] LustreError: 285441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787604793 with bad export cookie 1757421729986617584 [11361.295418] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11361.301779] LustreError: 285441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11361.589117] Lustre: server umount lustre-MDT0001 complete [11376.720827] Lustre: server umount lustre-OST0000 complete [11391.309547] Lustre: server umount lustre-OST0001 complete [11403.251476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11404.028274] LustreError: 290204:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11404.066181] LustreError: 290204:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [11410.814853] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11422.387692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11429.610579] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11440.913151] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11448.616343] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11456.720164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11465.310665] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11477.304768] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11497.385665] loop1: detected capacity change from 0 to 204800000 [11497.649992] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11497.656353] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11497.678070] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11503.467396] loop1: detected capacity change from 0 to 204800000 [11509.759420] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11509.916360] LustreError: 290191:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs9294: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11509.956550] LustreError: 290191:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs9294 [11509.965508] LustreError: 290191:0:(mgs_handler.c:517:mgs_target_reg()) fs9294-MDT0001: Error checking target index: rc = -13 [11509.989626] LustreError: fs9294-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11510.006612] LustreError: 293930:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11510.017970] LustreError: 293930:0:(tgt_mount.c:2132:server_put_super()) no obd fs9294-MDT0001 [11510.026560] LustreError: 293930:0:(tgt_mount.c:116:server_deregister_mount()) fs9294-MDT0001 not registered [11510.128685] Lustre: server umount fs9294-MDT0001 complete [11510.133981] LustreError: 293930:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11521.444635] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11521.703520] Lustre: Setting parameter fs9294-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs9294-MDT0001 [11521.846972] Lustre: srv-fs9294-MDT0001: No data found on store. Initialize space. [11521.854990] Lustre: Skipped 4 previous similar messages [11522.035859] Lustre: fs9294-MDT0001: new disk, initializing [11522.041573] Lustre: Skipped 3 previous similar messages [11522.301990] Lustre: srv-fs9294-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11522.324101] Lustre: 294398:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs9294-MDT0001-mdtlov: can't create new llog: rc = -115 [11522.349730] LustreError: 294398:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs9294-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11529.326083] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11538.238129] Lustre: server umount fs9294-MDT0001 complete [11544.050110] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11556.152750] loop1: detected capacity change from 0 to 204800000 [11562.754686] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11563.511883] Lustre: 295969:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs9294-MDT0001-mdtlov: can't create new llog: rc = -115 [11563.528248] LustreError: 295969:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs9294-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11569.799487] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11577.058528] Lustre: server umount fs9294-MDT0001 complete [11590.826857] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.59@tcp (stopping) [11590.838922] Lustre: Skipped 3 previous similar messages [11591.665341] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11591.679396] LustreError: Skipped 1 previous similar message [11591.692941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11596.912605] Lustre: server umount lustre-MDT0000 complete [11605.930172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11606.019261] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11610.974315] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11611.652164] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11619.388656] loop1: detected capacity change from 0 to 204800000 [11624.480634] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11624.937937] Lustre: 298417:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs9294-MDT0001-mdtlov: can't create new llog: rc = -115 [11624.948296] LustreError: 298417:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs9294-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11630.061506] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11635.226881] Lustre: server umount fs9294-MDT0001 complete [11666.914066] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11666.923529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11666.927067] Lustre: Skipped 5 previous similar messages [11671.573768] Lustre: server umount lustre-OST0000 complete [11682.578688] Lustre: server umount lustre-MDT0000 complete [11686.855990] LustreError: 290186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787605118 with bad export cookie 1757421729986620265 [11686.858705] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11686.864369] LustreError: 290186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11687.336926] Lustre: server umount lustre-MDT0001 complete [11702.546855] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [11707.130848] Key type lgssc unregistered [11707.691751] LNet: 300765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11707.696228] LNetError: 300765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11707.720471] LNet: Removed LNI 192.168.202.159@tcp [11709.275202] Key type .llcrypt unregistered [11709.277095] Key type ._llcrypt unregistered [11735.078996] Key type ._llcrypt registered [11735.081045] Key type .llcrypt registered [11735.213610] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [11736.967639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11737.000031] alg: No test for adler32 (adler32-zlib) [11738.137554] Lustre: Lustre: Build Version: 2.17.57_80_g3abb7b2 [11738.365297] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [11739.984177] Key type lgssc registered [11740.841787] Lustre: Echo OBD driver; http://www.lustre.org/ [11748.316575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11748.362649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11750.114690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11754.556593] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11760.619925] LustreError: 302040:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11760.652362] LustreError: 302040:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11765.150936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11765.493199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11769.717945] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11777.974271] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11783.708501] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11791.451403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11791.744186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11795.832349] LustreError: 303870:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11797.825791] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11801.067021] LustreError: 303869:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11805.380571] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11806.192082] LustreError: 303870:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11810.278604] LustreError: 303869:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11810.287397] LustreError: 303869:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11818.630955] LustreError: 304095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11818.677874] LustreError: 304095:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11826.155177] loop1: detected capacity change from 0 to 204800000 [11826.349672] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11826.356959] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11826.374181] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11831.890739] loop1: detected capacity change from 0 to 204800000 [11835.893370] LustreError: 304544:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11835.929647] LustreError: 304544:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [11837.888859] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11838.065839] Lustre: fs1550-OST0000: new disk, initializing [11838.069744] Lustre: srv-fs1550-OST0000: No data found on store. Initialize space. [11838.077650] Lustre: 305786:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11838.151878] Lustre: fs1550-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11847.433980] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11856.921289] Lustre: server umount fs1550-OST0000 complete [11869.830738] LustreError: 303869:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11869.858321] LustreError: 303869:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [11871.373967] loop1: detected capacity change from 0 to 204800000 [11875.824630] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11879.717029] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11879.953899] Lustre: fs1550-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11890.223893] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11891.207280] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11903.008534] loop2: detected capacity change from 0 to 204800000 [11903.527765] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11903.549905] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11903.587889] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11913.403931] loop2: detected capacity change from 0 to 204800000 [11921.179897] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11921.250857] LustreError: fs1550-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11921.263211] LustreError: fs1550-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11921.276497] LustreError: 308832:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11921.290598] LustreError: 308832:0:(tgt_mount.c:2132:server_put_super()) no obd fs1550-OST0001 [11921.302804] LustreError: 308832:0:(tgt_mount.c:116:server_deregister_mount()) fs1550-OST0001 not registered [11921.357528] Lustre: server umount fs1550-OST0001 complete [11921.360504] LustreError: 308832:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11935.687108] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11936.129243] Lustre: fs1550-OST0001: new disk, initializing [11936.141630] Lustre: srv-fs1550-OST0001: No data found on store. Initialize space. [11936.149501] Lustre: 309384:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11936.289702] Lustre: fs1550-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11936.447048] LustreError: 303870:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11936.490424] LustreError: 303870:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [11947.279120] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11958.465860] Lustre: server umount fs1550-OST0000 complete [11972.590462] Lustre: server umount fs1550-OST0001 complete [11988.591464] loop1: detected capacity change from 0 to 204800000 [11995.060214] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11995.264802] Lustre: fs1550-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12002.604840] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12010.337952] loop2: detected capacity change from 0 to 204800000 [12016.645592] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12023.762472] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12029.427512] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12034.161494] Lustre: server umount fs1550-OST0000 complete [12044.783831] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12044.895311] Lustre: server umount fs1550-OST0001 complete [12060.137695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12060.162614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12060.171668] Lustre: Skipped 1 previous similar message [12064.108618] Lustre: server umount lustre-OST0000 complete [12070.375985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12070.379773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12070.393379] Lustre: Skipped 2 previous similar messages [12070.411678] Lustre: Skipped 1 previous similar message [12075.489788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12075.499183] Lustre: Skipped 1 previous similar message [12080.613227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12080.635124] Lustre: Skipped 1 previous similar message [12084.192464] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12084.548504] Lustre: server umount lustre-MDT0000 complete [12085.734396] LustreError: 302041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12085.771566] LustreError: 302041:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 74 previous similar messages [12088.926052] LustreError: 303893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787605520 with bad export cookie 13475403204068624326 [12088.936276] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12088.949326] LustreError: 303893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12089.302565] Lustre: server umount lustre-MDT0001 complete [12101.609904] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [12104.575383] Key type lgssc unregistered [12104.880888] LNet: 315316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12104.891424] LNetError: 315316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12104.921259] LNet: Removed LNI 192.168.202.159@tcp [12106.039156] Key type .llcrypt unregistered [12106.042130] Key type ._llcrypt unregistered [12141.019714] Key type ._llcrypt registered [12141.023363] Key type .llcrypt registered [12141.159927] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [12159.014829] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [12160.242254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12160.443423] alg: No test for adler32 (adler32-zlib) [12161.631556] Lustre: Lustre: Build Version: 2.17.57_80_g3abb7b2 [12161.953364] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [12163.625045] Key type lgssc registered [12165.140797] Lustre: Echo OBD driver; http://www.lustre.org/ [12211.507546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12211.619186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12213.029159] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12213.095621] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12213.247892] Lustre: lustre-MDT0000: new disk, initializing [12213.382682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12213.417172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12218.524664] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12229.352705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12229.467858] Lustre: 319179:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12229.515413] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12229.519152] Lustre: Skipped 1 previous similar message [12229.551498] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12229.593581] Lustre: lustre-MDT0001: new disk, initializing [12229.686470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12229.735139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12229.755427] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12234.559762] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12246.850453] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12255.587995] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12263.254221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12263.523356] Lustre: lustre-OST0000: new disk, initializing [12263.531524] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12263.549876] Lustre: 321147:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12263.721196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12271.653227] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12272.746246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12272.762238] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12272.924472] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12283.185940] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12285.784750] Lustre: Mounted lustre-client - version 2.17.57_80_g3abb7b2 [12286.989661] Lustre: Unmounted lustre-client [12298.722067] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12298.723165] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12298.726272] LustreError: Skipped 1 previous similar message [12298.727105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12298.753986] Lustre: Skipped 1 previous similar message [12301.282919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12301.295125] Lustre: Skipped 2 previous similar messages [12304.261818] Lustre: server umount lustre-OST0000 complete [12311.529555] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12311.541526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12311.553838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12316.646996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12316.656922] Lustre: Skipped 2 previous similar messages [12323.302552] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12324.194366] Lustre: server umount lustre-MDT0000 complete [12328.824462] LustreError: 319171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787605760 with bad export cookie 4168948372150197567 [12328.846524] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12329.263240] Lustre: server umount lustre-MDT0001 complete [12344.295259] LNet: 1 local NIs in recovery (showing 1): 192.168.202.159@tcp [12345.401421] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [12348.926392] Key type lgssc unregistered [12349.212906] LNet: 323314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12349.224669] LNetError: 323314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12416.813242] LNet: Removed LNI 192.168.202.159@tcp [12417.920118] Key type .llcrypt unregistered [12417.922160] Key type ._llcrypt unregistered [12438.340713] Key type ._llcrypt registered [12438.345555] Key type .llcrypt registered [12438.459995] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [12455.546443] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [12456.260727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12456.299065] alg: No test for adler32 (adler32-zlib) [12457.400151] Lustre: Lustre: Build Version: 2.17.57_80_g3abb7b2 [12457.692671] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [12459.377189] Key type lgssc registered [12460.363574] Lustre: Echo OBD driver; http://www.lustre.org/ [12492.057761] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 17:12:02 (1787605922) [12499.955277] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12499.992636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12501.239529] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12501.262499] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12501.333872] Lustre: lustre-MDT0000: new disk, initializing [12501.394258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12501.408115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12505.779773] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12515.501772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12515.663487] Lustre: 326538:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12515.745751] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12515.751417] Lustre: Skipped 1 previous similar message [12515.795613] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12515.930437] Lustre: lustre-MDT0001: new disk, initializing [12516.174659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12516.234242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12516.246330] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12520.721946] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12530.268382] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12536.460932] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12544.385305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12544.760713] Lustre: lustre-OST0000: new disk, initializing [12544.764990] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12544.772401] Lustre: 328504:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12544.872064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12551.238823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12551.256398] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12551.401417] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12552.187757] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12563.582202] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12582.374070] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12582.375352] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12582.393390] LustreError: Skipped 1 previous similar message [12582.401494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12582.411150] Lustre: Skipped 1 previous similar message [12582.427417] Lustre: Skipped 1 previous similar message [12584.218670] Lustre: server umount lustre-OST0000 complete [12592.616852] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12592.629314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12592.636985] Lustre: Skipped 1 previous similar message [12592.647709] Lustre: Skipped 1 previous similar message [12594.492807] Lustre: server umount lustre-MDT0000 complete [12598.454497] LustreError: 326531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787606030 with bad export cookie 15991020155318251589 [12598.456957] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12598.466306] LustreError: 326531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12598.796448] Lustre: server umount lustre-MDT0001 complete [12610.488414] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [12613.010990] Key type lgssc unregistered [12613.402834] LNet: 330415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12613.423143] LNetError: 330415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12613.449690] LNet: Removed LNI 192.168.202.159@tcp [12614.146980] Key type .llcrypt unregistered [12614.151842] Key type ._llcrypt unregistered [12640.573539] Key type ._llcrypt registered [12640.579211] Key type .llcrypt registered [12640.670960] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [12642.215612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12642.280995] alg: No test for adler32 (adler32-zlib) [12643.271889] Lustre: Lustre: Build Version: 2.17.57_80_g3abb7b2 [12643.491564] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [12645.145323] Key type lgssc registered [12646.438939] Lustre: Echo OBD driver; http://www.lustre.org/ [12653.132705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12653.145796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12654.504802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12659.741700] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12664.806961] LustreError: 331690:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12664.821161] LustreError: 331690:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12668.711560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12669.156462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12673.501441] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12682.582790] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12691.680457] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12702.941868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12703.770658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12713.516974] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12723.263984] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12735.352788] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12737.434455] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12744.684159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12744.692628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12744.696452] Lustre: Skipped 2 previous similar messages [12744.711920] Lustre: Skipped 2 previous similar messages [12746.400967] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.59@tcp (stopping) [12748.764970] Lustre: server umount lustre-MDT0000 complete [12749.796229] LustreError: 331690:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12749.827111] LustreError: 331690:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12751.487139] LustreError: 331686:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12752.856471] LustreError: 331670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787606184 with bad export cookie 17757284580392861604 [12752.877278] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12752.897580] LustreError: 331670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12754.913439] LustreError: 331685:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12754.915657] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12754.941238] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12757.922117] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12757.935632] Lustre: Skipped 1 previous similar message [12759.314353] Lustre: server umount lustre-MDT0001 complete [12763.942925] Lustre: server umount lustre-OST0000 complete [12773.522239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12778.834712] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12782.153535] Lustre: 335880:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.159@tcp [12784.512672] Lustre: server umount MGS complete [12794.071743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12794.487525] LustreError: 336451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12794.516274] LustreError: 336451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12794.620492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12799.523869] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12802.684743] LustreError: 336448:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12802.699437] LustreError: 336448:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12809.126427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12809.545415] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12814.226712] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12823.849532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12824.550446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12831.852311] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12839.993615] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12842.167430] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12849.022203] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12851.354508] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12861.416425] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12861.429572] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12861.435986] LustreError: Skipped 1 previous similar message [12861.454209] Lustre: Skipped 1 previous similar message [12861.461153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12864.752770] Lustre: server umount lustre-OST0000 complete [12871.140192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12871.155665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12871.184087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12871.192036] Lustre: Skipped 1 previous similar message [12874.958603] Lustre: server umount lustre-MDT0000 complete [12876.781271] LustreError: 336448:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12876.800701] LustreError: 336448:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12879.812828] LustreError: 336476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787606311 with bad export cookie 17757284580392862444 [12879.816586] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12879.830211] LustreError: 336476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12880.426338] Lustre: server umount lustre-MDT0001 complete [12892.297435] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [12895.267680] Key type lgssc unregistered [12895.583827] LNet: 339893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12895.590335] LNetError: 339893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12895.607273] LNet: Removed LNI 192.168.202.159@tcp [12896.520578] Key type .llcrypt unregistered [12896.522111] Key type ._llcrypt unregistered [12913.846890] Key type ._llcrypt registered [12913.848663] Key type .llcrypt registered [12913.942246] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [12930.882966] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [12932.003592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12932.025093] alg: No test for adler32 (adler32-zlib) [12933.202582] Lustre: Lustre: Build Version: 2.17.57_80_g3abb7b2 [12933.418531] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [12935.096264] Key type lgssc registered [12936.142626] Lustre: Echo OBD driver; http://www.lustre.org/ [12972.750981] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12972.788356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12974.038836] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12974.066925] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12974.139530] Lustre: lustre-MDT0000: new disk, initializing [12974.203718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12974.233329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12978.171503] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12989.409835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12989.480402] Lustre: 343053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12989.530327] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12989.535086] Lustre: Skipped 1 previous similar message [12989.586968] Lustre: lustre-MDT0001: new disk, initializing [12989.678760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12989.707537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12989.718229] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12993.877497] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [13002.905541] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13008.944465] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13016.307791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13016.612290] Lustre: lustre-OST0000: new disk, initializing [13016.617231] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13016.624935] Lustre: 345024:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13016.731246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13022.443384] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [13025.436525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13025.455492] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13025.692942] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13031.361237] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13035.839318] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13036.018196] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13040.245863] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13040.492555] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13044.199112] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13044.230402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13048.726346] Lustre: server umount lustre-OST0000 complete [13054.435529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13054.441187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13054.448841] Lustre: Skipped 2 previous similar messages [13054.469135] Lustre: Skipped 2 previous similar messages [13059.554624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13059.563038] Lustre: Skipped 1 previous similar message [13064.675795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13064.682984] Lustre: Skipped 1 previous similar message [13067.233125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13067.566976] Lustre: server umount lustre-MDT0000 complete [13071.246460] LustreError: 343851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787606503 with bad export cookie 10627561645648750066 [13071.247024] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13071.267778] LustreError: 343851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13071.592836] Lustre: server umount lustre-MDT0001 complete [13080.221662] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 17:21:50 (1787606510) [13100.012237] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [13114.083931] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [13154.698462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13154.947049] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13154.967551] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13155.029534] Lustre: lustre-MDT0000: new disk, initializing [13155.095728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13155.107412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13158.991279] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [13169.631504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13169.866562] Lustre: 349805:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13169.974205] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13169.985729] Lustre: Skipped 1 previous similar message [13170.112096] Lustre: lustre-MDT0001: new disk, initializing [13170.259601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13170.308244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13170.323599] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13174.994198] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [13185.055644] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13191.728452] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13200.585758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13201.038936] Lustre: lustre-OST0000: new disk, initializing [13201.043481] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13201.051325] Lustre: 351773:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13201.162580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13209.922182] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [13210.231337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13210.244565] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13210.494923] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13222.126452] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13261.795942] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13261.797212] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13261.810083] LustreError: Skipped 1 previous similar message [13261.813884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13261.824032] Lustre: Skipped 1 previous similar message [13261.850535] Lustre: Skipped 1 previous similar message [13266.299484] Lustre: server umount lustre-OST0000 complete [13272.033632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13272.035074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13272.039682] Lustre: Skipped 1 previous similar message [13272.053267] Lustre: Skipped 1 previous similar message [13276.589982] Lustre: server umount lustre-MDT0000 complete [13277.154275] LustreError: 349817:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13277.174921] LustreError: 349817:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13280.216060] LustreError: 349795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787606712 with bad export cookie 10627561645648750948 [13280.218110] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13280.233365] LustreError: 349795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13280.493869] Lustre: server umount lustre-MDT0001 complete [13291.847605] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [13294.960558] Key type lgssc unregistered [13295.353801] LNet: 353949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13295.358301] LNetError: 353949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13295.377398] LNet: Removed LNI 192.168.202.159@tcp [13296.196520] Key type .llcrypt unregistered [13296.202583] Key type ._llcrypt unregistered [13392.892960] Key type ._llcrypt registered [13392.894670] Key type .llcrypt registered [13393.066971] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [13393.380523] Key type .llcrypt unregistered [13393.386470] Key type ._llcrypt unregistered [13480.033708] Key type ._llcrypt registered [13480.035876] Key type .llcrypt registered [13480.134313] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [13480.461397] Key type .llcrypt unregistered [13480.463232] Key type ._llcrypt unregistered [13512.503989] Key type ._llcrypt registered [13512.505815] Key type .llcrypt registered [13512.584739] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [13512.864661] Key type .llcrypt unregistered [13512.866882] Key type ._llcrypt unregistered [13546.270765] Key type ._llcrypt registered [13546.273026] Key type .llcrypt registered [13546.369175] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [13546.653411] Key type .llcrypt unregistered [13546.655383] Key type ._llcrypt unregistered [13633.292683] Key type ._llcrypt registered [13633.294428] Key type .llcrypt registered [13633.375224] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [13633.680335] Key type .llcrypt unregistered [13633.683227] Key type ._llcrypt unregistered [13670.542417] Key type ._llcrypt registered [13670.544581] Key type .llcrypt registered [13670.717392] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [13671.311609] Key type .llcrypt unregistered [13671.323537] Key type ._llcrypt unregistered [13703.770746] Key type ._llcrypt registered [13703.775917] Key type .llcrypt registered [13703.899973] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [13704.315447] Key type .llcrypt unregistered [13704.317138] Key type ._llcrypt unregistered [13732.999970] Key type ._llcrypt registered [13733.001796] Key type .llcrypt registered [13733.093268] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [13733.463422] Key type .llcrypt unregistered [13733.465672] Key type ._llcrypt unregistered [13762.834716] Key type ._llcrypt registered [13762.839868] Key type .llcrypt registered [13762.963283] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [13763.405416] Key type .llcrypt unregistered [13763.408307] Key type ._llcrypt unregistered [13793.880124] Key type ._llcrypt registered [13793.881857] Key type .llcrypt registered [13794.006538] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [13794.325539] Key type .llcrypt unregistered [13794.331380] Key type ._llcrypt unregistered [13859.236256] Key type ._llcrypt registered [13859.240409] Key type .llcrypt registered [13859.365189] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [13859.894855] Key type .llcrypt unregistered [13859.897909] Key type ._llcrypt unregistered [13890.763747] Key type ._llcrypt registered [13890.767100] Key type .llcrypt registered [13890.952716] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [13891.294424] Key type .llcrypt unregistered [13891.297630] Key type ._llcrypt unregistered [13923.843755] Key type ._llcrypt registered [13923.846419] Key type .llcrypt registered [13923.944288] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [13924.222627] Key type .llcrypt unregistered [13924.224561] Key type ._llcrypt unregistered [13982.540548] Key type ._llcrypt registered [13982.548476] Key type .llcrypt registered [13982.681956] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [13999.616312] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [14001.442388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14001.513577] alg: No test for adler32 (adler32-zlib) [14003.280836] Lustre: Lustre: Build Version: 2.17.57_80_g3abb7b2 [14003.703985] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [14005.536213] Key type lgssc registered [14006.673575] Lustre: Echo OBD driver; http://www.lustre.org/ [14042.483969] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14042.513221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14043.761738] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14043.820904] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14043.909279] Lustre: tst250fs-MDT0000: new disk, initializing [14043.967560] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14044.005914] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14048.557775] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [14060.918925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14061.075824] Lustre: 374332:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14061.151509] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14061.155493] Lustre: Skipped 1 previous similar message [14061.227574] Lustre: tst250fs-MDT0001: new disk, initializing [14061.328619] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14061.385372] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14061.401311] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14066.989996] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [14073.147405] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14081.031802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14081.375718] Lustre: tst250fs-OST0000: new disk, initializing [14081.381505] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14081.393823] Lustre: 375964:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14081.477857] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14087.821794] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14087.844994] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14088.029869] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14090.250541] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [14101.199885] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14101.413657] Lustre: tst250fs-OST0001: new disk, initializing [14101.417452] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14101.424594] Lustre: 376838:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14101.518156] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14109.312184] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14109.322255] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14109.441428] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [14109.476187] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14115.768701] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14117.847991] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [14117.860277] Lustre: Skipped 1 previous similar message [14138.337782] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14138.346466] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14138.364161] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14140.392443] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14140.408818] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14140.420607] Lustre: Skipped 1 previous similar message [14143.999493] Lustre: server umount tst250fs-MDT0000 complete [14148.258640] LustreError: 376837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787607580 with bad export cookie 1694787986215528776 [14148.271454] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14148.280966] LustreError: 376837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14148.896418] Lustre: server umount tst250fs-MDT0001 complete [14163.802752] Lustre: server umount tst250fs-OST0000 complete [14166.818966] Lustre: 372555:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787607582/real 1787607582] req@ffff8e1b8cc71500 x1874442253836160/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787607598 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14166.872108] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14166.896614] Lustre: Skipped 1 previous similar message [14168.113357] Lustre: server umount tst250fs-OST0001 complete [14186.079475] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [14193.233636] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [14226.329874] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 17:40:56 (1787607656) [14228.167357] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14248.095208] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [14258.825727] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [14293.539421] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13947 sec ========= 17:42:03 (1787607723) [14295.809560] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 17:42:06 (1787607726) === [14299.999435] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 17:42:10 (1787607730) === [14347.692208] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [14351.415411] Key type lgssc unregistered [14351.843074] LNet: 385988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14351.848633] LNetError: 385988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14351.867858] LNet: Removed LNI 192.168.202.159@tcp [14352.738224] Key type .llcrypt unregistered [14352.741066] Key type ._llcrypt unregistered