[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 505575184 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003205] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008610] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009033] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010023] pid_max: default: 32768 minimum: 301 [ 0.011160] LSM: Security Framework initializing [ 0.012069] Yama: becoming mindful. [ 0.013050] SELinux: Initializing. [ 0.014066] *** VALIDATE selinux *** [ 0.022274] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027063] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028196] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029128] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031010] *** VALIDATE tmpfs *** [ 0.032505] *** VALIDATE proc *** [ 0.033211] *** VALIDATE cgroup *** [ 0.034007] *** VALIDATE cgroup2 *** [ 0.035277] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036190] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037015] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038042] Spectre V2 : User space: Vulnerable [ 0.039012] Speculative Store Bypass: Vulnerable [ 0.042391] debug: unmapping init [mem 0xffffffff84c59000-0xffffffff84c60fff] [ 0.045000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045832] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046034] ... version: 2 [ 0.047016] ... bit width: 48 [ 0.048017] ... generic registers: 4 [ 0.049017] ... value mask: 0000ffffffffffff [ 0.050020] ... max period: 00007fffffffffff [ 0.051012] ... fixed-purpose events: 3 [ 0.052011] ... event mask: 000000070000000f [ 0.054310] rcu: Hierarchical SRCU implementation. [ 0.056863] smp: Bringing up secondary CPUs ... [ 0.057681] x86: Booting SMP configuration: [ 0.058037] .... node #0, CPUs: #1 #2 #3 [ 0.061708] smp: Brought up 1 node, 4 CPUs [ 0.063021] smpboot: Max logical packages: 1 [ 0.064030] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.265227] node 0 deferred pages initialised in 199ms [ 0.269017] devtmpfs: initialized [ 0.270360] x86/mm: Memory block size: 128MB [ 0.273000] gcov: version magic: 0x41383552 [ 0.275328] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.279119] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.281365] pinctrl core: initialized pinctrl subsystem [ 0.283169] [ 0.283714] ************************************************************* [ 0.285016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.287017] ** ** [ 0.290016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.292013] ** ** [ 0.294015] ** This means that this kernel is built to expose internal ** [ 0.296015] ** IOMMU data structures, which may compromise security on ** [ 0.298017] ** your system. ** [ 0.300028] ** ** [ 0.302016] ** If you see this message and you are not debugging the ** [ 0.304014] ** kernel, report this immediately to your vendor! ** [ 0.306016] ** ** [ 0.309015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.311013] ************************************************************* [ 0.313760] NET: Registered protocol family 16 [ 0.315400] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.318084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.320101] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.324049] cpuidle: using governor menu [ 0.325769] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.328664] PCI: Using configuration type 1 for base access [ 0.330158] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.341072] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.342037] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.344097] cryptd: max_cpu_qlen set to 1000 [ 0.347136] ACPI: Added _OSI(Module Device) [ 0.348014] ACPI: Added _OSI(Processor Device) [ 0.349018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.351023] ACPI: Added _OSI(Processor Aggregator Device) [ 0.356406] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.364153] ACPI: Interpreter enabled [ 0.365077] ACPI: PM: (supports S0 S3 S4 S5) [ 0.367022] ACPI: Using IOAPIC for interrupt routing [ 0.368162] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.372498] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.382855] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.384062] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.387027] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.390102] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.395561] acpiphp: Slot [2] registered [ 0.396144] acpiphp: Slot [5] registered [ 0.398142] acpiphp: Slot [6] registered [ 0.399183] acpiphp: Slot [7] registered [ 0.400178] acpiphp: Slot [8] registered [ 0.402143] acpiphp: Slot [9] registered [ 0.403176] acpiphp: Slot [10] registered [ 0.405191] acpiphp: Slot [3] registered [ 0.406151] acpiphp: Slot [4] registered [ 0.408158] acpiphp: Slot [11] registered [ 0.409133] acpiphp: Slot [12] registered [ 0.411138] acpiphp: Slot [13] registered [ 0.412121] acpiphp: Slot [14] registered [ 0.414089] acpiphp: Slot [15] registered [ 0.415100] acpiphp: Slot [16] registered [ 0.416165] acpiphp: Slot [17] registered [ 0.418161] acpiphp: Slot [18] registered [ 0.419121] acpiphp: Slot [19] registered [ 0.421142] acpiphp: Slot [20] registered [ 0.422133] acpiphp: Slot [21] registered [ 0.424135] acpiphp: Slot [22] registered [ 0.425120] acpiphp: Slot [23] registered [ 0.427165] acpiphp: Slot [24] registered [ 0.428122] acpiphp: Slot [25] registered [ 0.429121] acpiphp: Slot [26] registered [ 0.431122] acpiphp: Slot [27] registered [ 0.432124] acpiphp: Slot [28] registered [ 0.433131] acpiphp: Slot [29] registered [ 0.435169] acpiphp: Slot [30] registered [ 0.436144] acpiphp: Slot [31] registered [ 0.438081] PCI host bridge to bus 0000:00 [ 0.439029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.442031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.443021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.445031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.448031] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.451037] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.452201] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.456065] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.459340] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.468023] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.473492] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.476026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.478024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.480024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.483585] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.486804] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.489050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.493325] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.499020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.512020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.518020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.524779] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.531026] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.537022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.556032] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.568915] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.586020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.595032] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.617025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.634146] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.641022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.648022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.661032] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.671705] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.689021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.702128] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.724024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.733387] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.738019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.744022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.763025] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.780277] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.799021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.808022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.822023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.833918] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.836413] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.838405] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.841414] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.845307] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.851036] iommu: Default domain type: Passthrough [ 0.853457] SCSI subsystem initialized [ 0.854000] ACPI: bus type USB registered [ 0.856129] usbcore: registered new interface driver usbfs [ 0.858081] usbcore: registered new interface driver hub [ 0.859092] usbcore: registered new device driver usb [ 0.861216] pps_core: LinuxPPS API ver. 1 registered [ 0.863016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.866079] PTP clock support registered [ 0.868165] EDAC MC: Ver: 3.0.0 [ 0.870189] PCI: Using ACPI for IRQ routing [ 0.872971] NetLabel: Initializing [ 0.874014] NetLabel: domain hash size = 128 [ 0.875013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.878095] NetLabel: unlabeled traffic allowed by default [ 0.880125] vgaarb: loaded [ 0.881269] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.882009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.886017] clocksource: Switched to clocksource kvm-clock [ 0.992036] VFS: Disk quotas dquot_6.6.0 [ 0.993074] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.995524] *** VALIDATE ramfs *** [ 0.996912] *** VALIDATE hugetlbfs *** [ 0.998662] pnp: PnP ACPI init [ 1.001232] pnp: PnP ACPI: found 6 devices [ 1.021993] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.023905] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.025231] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.026835] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.028600] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.030860] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.033126] NET: Registered protocol family 2 [ 1.035207] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.039672] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.043839] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.047535] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.050257] TCP: Hash tables configured (established 65536 bind 65536) [ 1.052974] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.055861] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.058395] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.061230] NET: Registered protocol family 1 [ 1.063637] RPC: Registered named UNIX socket transport module. [ 1.066574] RPC: Registered udp transport module. [ 1.068028] RPC: Registered tcp transport module. [ 1.069658] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.071623] NET: Registered protocol family 44 [ 1.073073] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.075289] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.077732] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.080609] PCI: CLS 0 bytes, default 64 [ 1.082561] Unpacking initramfs... [ 2.488702] debug: unmapping init [mem 0xffff9a107cc54000-0xffff9a107ffbffff] [ 2.492527] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.495016] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.497878] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.062462] Initialise system trusted keyrings [ 3.064110] Key type blacklist registered [ 3.066504] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.083487] zbud: loaded [ 3.088507] *** VALIDATE nfs *** [ 3.090464] *** VALIDATE nfs4 *** [ 3.092430] pstore: using deflate compression [ 3.100289] Platform Keyring initialized [ 3.316834] NET: Registered protocol family 38 [ 3.318933] Key type asymmetric registered [ 3.320508] Asymmetric key parser 'x509' registered [ 3.322294] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.325171] io scheduler mq-deadline registered [ 3.326310] io scheduler kyber registered [ 3.327410] io scheduler bfq registered [ 3.329270] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.332306] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.335352] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.338241] ACPI: Power Button [PWRF] [ 3.343741] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.351408] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.368873] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.380321] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.402468] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.437440] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.473327] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.480559] Non-volatile memory driver v1.3 [ 3.483653] Linux agpgart interface v0.103 [ 3.558222] virtio_blk virtio1: [vda] 146000 512-byte logical blocks (74.8 MB/71.3 MiB) [ 3.561957] vda: detected capacity change from 0 to 74752000 [ 3.587521] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.590968] vdb: detected capacity change from 0 to 1073741824 [ 3.608767] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.612132] vdc: detected capacity change from 0 to 2621440000 [ 3.635353] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.639438] vdd: detected capacity change from 0 to 2621440000 [ 3.656991] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.661562] vde: detected capacity change from 0 to 4294967296 [ 3.679493] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.683017] vdf: detected capacity change from 0 to 4294967296 [ 3.699663] libphy: Fixed MDIO Bus: probed [ 3.714329] usbcore: registered new interface driver usbserial_generic [ 3.717616] usbserial: USB Serial support registered for generic [ 3.719637] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.726023] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.727457] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.729566] mousedev: PS/2 mouse device common for all mice [ 3.732484] rtc_cmos 00:05: RTC can wake from S4 [ 3.734575] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.737223] rtc_cmos 00:05: registered as rtc0 [ 3.742753] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.747125] intel_pstate: CPU model not supported [ 3.749464] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.756036] hid: raw HID events driver (C) Jiri Kosina [ 3.757090] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.758913] usbcore: registered new interface driver usbhid [ 3.763138] usbhid: USB HID core driver [ 3.764525] drop_monitor: Initializing network drop monitor service [ 3.766964] Initializing XFRM netlink socket [ 3.768312] NET: Registered protocol family 10 [ 3.770509] Segment Routing with IPv6 [ 3.771485] NET: Registered protocol family 17 [ 3.773611] mpls_gso: MPLS GSO support [ 3.778458] RAS: Correctable Errors collector initialized. [ 3.780457] AVX version of gcm_enc/dec engaged. [ 3.781594] AES CTR mode by8 optimization enabled [ 3.909805] sched_clock: Marking stable (3909648256, 0)->(4798940054, -889291798) [ 3.917189] registered taskstats version 1 [ 3.919610] Loading compiled-in X.509 certificates [ 3.922980] zswap: loaded using pool lzo/zbud [ 4.009823] Key type big_key registered [ 4.053696] Key type encrypted registered [ 4.055335] ima: No TPM chip found, activating TPM-bypass! [ 4.061606] ima: Allocated hash algorithm: sha1 [ 4.063452] ima: No architecture policies found [ 4.069528] evm: Initialising EVM extended attributes: [ 4.072627] evm: security.selinux [ 4.077542] evm: security.ima [ 4.079472] evm: security.capability [ 4.081313] evm: HMAC attrs: 0x1 [ 4.084946] rtc_cmos 00:05: setting system clock to 2026-08-19 16:11:16 UTC (1787155876) [ 4.104908] debug: unmapping init [mem 0xffffffff85c03000-0xffffffff85dfffff] [ 4.115985] debug: unmapping init [mem 0xffffffff84982000-0xffffffff84c58fff] [ 4.135214] Write protecting the kernel read-only data: 28672k [ 4.155265] debug: unmapping init [mem 0xffffffff83003000-0xffffffff831fffff] [ 4.162417] debug: unmapping init [mem 0xffffffff83914000-0xffffffff839fffff] [ 4.232215] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.265289] systemd[1]: Detected virtualization kvm. [ 4.274406] systemd[1]: Detected architecture x86-64. [ 4.278427] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.365610] systemd[1]: No hostname configured. [ 4.368690] systemd[1]: Set hostname to . [ 4.379920] random: systemd: uninitialized urandom read (16 bytes read) [ 4.391787] systemd[1]: Initializing machine ID from random generator. [ 4.562970] random: ln: uninitialized urandom read (6 bytes read) [ 4.764673] random: systemd: uninitialized urandom read (16 bytes read) [ 4.769831] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.777448] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.784749] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.276989] device-mapper: uevent: version 1.0.3 [ 6.281290] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 8.055794] random: fast init done [ 8.063616] virtio_net virtio0 ens2: renamed from eth0 [ 8.262433] scsi host0: ata_piix [ 8.351648] scsi host1: ata_piix [ 8.357303] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.369138] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.599876] random: crng init done [ 13.601522] random: 7 urandom warning(s) missed due to ratelimiting [ 14.386107] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ 15.944034] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Reached target Remote File Systems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.567980] printk: systemd: 26 output lines suppressed due to ratelimiting [ 19.197577] SELinux: Disabled at runtime. [ 19.308584] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 19.323151] systemd[1]: Detected virtualization kvm. [ 19.327217] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 20.907651] systemd[1]: initrd-switch-root.service: Succeeded. [ 20.918289] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 20.926223] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 20.935955] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 20.941249] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 20.967863] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.003662] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ 21.295848] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 22.186362] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 22.916440] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 23.026546] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 23.347390] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 23.409092] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (7s / no limit)[ 28.679053] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (8s / no limit)[ 29.397899] NFS: Registering the id_resolver key type [ 29.399960] Key type id_resolver registered [ 29.401855] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg215-server login: [ 68.422053] hrtimer: interrupt took 2055773 ns [ 83.675833] libcfs: loading out-of-tree module taints kernel. [ 83.730136] Key type ._llcrypt registered [ 83.732203] Key type .llcrypt registered [ 83.848798] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 102.040524] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 103.707574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 103.722967] alg: No test for adler32 (adler32-zlib) [ 105.286326] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 106.165838] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 108.016697] Key type lgssc registered [ 109.872892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 129.039714] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 172.435284] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 185.434436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 185.476909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 186.881176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 186.950309] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 187.080988] Lustre: lustre-MDT0000: new disk, initializing [ 187.205451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 187.267378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 192.042553] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 205.741676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 205.899127] Lustre: 6515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 205.975454] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 205.985138] Lustre: Skipped 1 previous similar message [ 206.060552] Lustre: lustre-MDT0001: new disk, initializing [ 206.120311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 206.140297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 206.151469] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 210.513741] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.300037] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 225.751805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 226.164805] Lustre: lustre-OST0000: new disk, initializing [ 226.168277] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 226.174934] Lustre: 8452:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 226.240347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 230.983764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 230.991491] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 231.150152] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 232.637759] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 249.684723] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 249.818544] Lustre: lustre-OST0001: new disk, initializing [ 249.822362] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 249.827788] Lustre: 9526:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 249.893931] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 255.956471] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 258.637077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 258.660915] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 258.738765] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 267.005464] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 272.407635] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 278.699780] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing check_logdir /tmp/testlogs/ [ 284.193323] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing yml_node [ 289.176632] Lustre: DEBUG MARKER: Client: 2.17.56.3 [ 292.198425] Lustre: DEBUG MARKER: MDS: 2.17.56.3 [ 294.833585] Lustre: DEBUG MARKER: OSS: 2.17.56.3 [ 296.324275] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 19 12:16:08 EDT 2026 [ 311.128479] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 312.647679] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 329.185151] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 329.197064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 329.210828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 330.726524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 330.727776] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 334.400716] Lustre: server umount lustre-MDT0000 complete [ 337.841929] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787156210 with bad export cookie 13394195026850859411 [ 337.854525] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 340.965818] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 340.967208] LustreError: 8459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 340.983907] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 340.983928] Lustre: Skipped 3 previous similar messages [ 340.991033] Lustre: Skipped 3 previous similar messages [ 341.042315] LustreError: 8459:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 343.010752] LustreError: 6522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.016378] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 343.058923] Lustre: Skipped 2 previous similar messages [ 348.130863] LustreError: 6523:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 348.134943] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 348.156033] LustreError: 6523:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 348.170802] Lustre: Skipped 1 previous similar message [ 352.226051] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 352.740610] Lustre: server umount lustre-MDT0001 complete [ 357.701847] Lustre: server umount lustre-OST0000 complete [ 362.002038] Lustre: server umount lustre-OST0001 complete [ 368.764476] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 379.232249] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 414.470470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 414.796840] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 414.822814] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 414.918147] Lustre: lustre-MDT0000: new disk, initializing [ 415.009142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 415.038618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 418.865428] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 428.902311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 429.051444] Lustre: 15077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 429.105122] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 429.108812] Lustre: Skipped 1 previous similar message [ 429.208353] Lustre: lustre-MDT0001: new disk, initializing [ 429.289477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 429.323133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 429.337843] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 433.949896] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 442.954523] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 448.894889] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 455.716588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 456.075950] Lustre: lustre-OST0000: new disk, initializing [ 456.081881] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 456.103128] Lustre: 17045:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 456.214595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 457.809186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 457.818256] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 457.942984] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 462.548613] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 471.803709] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 476.499502] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 476.711904] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 482.213471] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 482.540930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 488.947746] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 488.949247] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 488.954796] LustreError: Skipped 1 previous similar message [ 488.968965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 488.980560] Lustre: Skipped 1 previous similar message [ 488.996842] Lustre: Skipped 1 previous similar message [ 490.866585] Lustre: server umount lustre-OST0000 complete [ 496.097584] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 496.108323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 501.927724] Lustre: server umount lustre-MDT0000 complete [ 504.296042] LustreError: 15084:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 504.332970] LustreError: 15084:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 507.183320] LustreError: 15068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787156379 with bad export cookie 13394195026850860944 [ 507.206689] LustreError: 15068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 507.210678] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 507.949817] Lustre: server umount lustre-MDT0001 complete [ 514.875537] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 12:19:45 (1787156385) [ 540.547844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 540.924862] LustreError: 19920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.048257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 546.769435] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 551.400531] LustreError: 19921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 551.418352] LustreError: 19921:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 555.563734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 561.231796] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 570.055742] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 577.098626] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 579.566531] Lustre: Failing over lustre-MDT0000 [ 579.955421] Lustre: server umount lustre-MDT0000 complete [ 580.645028] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 580.662038] Lustre: Skipped 1 previous similar message [ 580.667025] LustreError: 19915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 580.681393] LustreError: 19915:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 596.961147] Lustre: 3647:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787156453/real 1787156453] req@ffff9a0fc2253480 x1873968866943616/t0(0) o400->MGC192.168.202.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787156469 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 596.988461] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 599.145972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 608.228357] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb9e1bd6953bfd516 [ 608.242442] Lustre: MGC192.168.202.115@tcp: Connection restored to 0@lo (at 0@lo) [ 608.528466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 608.539640] Lustre: Skipped 1 previous similar message [ 608.602240] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 612.613763] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 613.863379] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 613.871560] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 613.891785] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 622.156327] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 629.217380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 629.224584] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 629.233519] Lustre: Skipped 1 previous similar message [ 629.237034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 629.239835] Lustre: Skipped 3 previous similar messages [ 632.846145] Lustre: server umount lustre-MDT0000 complete [ 636.408267] LustreError: 22583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787156508 with bad export cookie 13394195026850862358 [ 636.419661] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 636.426456] LustreError: 22583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 636.841980] Lustre: server umount lustre-MDT0001 complete [ 646.160961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 646.749570] LustreError: 23260:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 646.767718] LustreError: 23260:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 16 previous similar messages [ 652.116395] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 661.644928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 667.877457] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 676.749428] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 683.245345] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 685.694117] Lustre: Failing over lustre-MDT0000 [ 685.885263] Lustre: server umount lustre-MDT0000 complete [ 686.764649] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 686.790555] Lustre: Skipped 2 previous similar messages [ 703.328122] Lustre: 3649:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787156559/real 1787156559] req@ffff9a0fc3536680 x1873968866973184/t0(0) o400->MGC192.168.202.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787156575 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 703.356442] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 706.267405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 712.615239] LustreError: 23989:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 712.656726] LustreError: 23989:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 712.901963] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 717.900680] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 718.311090] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 718.314642] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 718.324824] Lustre: Skipped 1 previous similar message [ 718.349921] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 728.426378] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 733.666415] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 733.679658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 733.686258] Lustre: Skipped 2 previous similar messages [ 739.157845] Lustre: server umount lustre-MDT0000 complete [ 743.581231] LustreError: 23997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787156615 with bad export cookie 13394195026850863296 [ 743.593888] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 743.925710] Lustre: server umount lustre-MDT0001 complete [ 753.714975] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 12:23:45 (1787156625) [ 769.359389] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 783.392576] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 835.204826] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 848.689267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 849.130572] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 849.175729] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 849.263676] Lustre: lustre-MDT0000: new disk, initializing [ 849.348469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 849.350750] Lustre: Skipped 3 previous similar messages [ 849.369211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 854.454580] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 868.350292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 868.504784] Lustre: 29676:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 869.058648] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 876.030306] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 889.290952] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 897.646830] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 904.910216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 905.389975] Lustre: 31651:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 907.449430] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 907.483830] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 912.908153] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 912.937598] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 912.941432] Lustre: Skipped 1 previous similar message [ 912.950832] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 913.038312] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 923.697164] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 930.104455] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 953.829022] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 953.835593] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 953.841751] LustreError: Skipped 1 previous similar message [ 953.856075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 953.861585] Lustre: Skipped 3 previous similar messages [ 953.872629] Lustre: Skipped 3 previous similar messages [ 965.091527] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 965.419952] Lustre: server umount lustre-OST0000 complete [ 975.592398] Lustre: server umount lustre-MDT0000 complete [ 979.425042] LustreError: 29689:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 979.440755] LustreError: 29689:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 979.871386] LustreError: 29670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787156852 with bad export cookie 13394195026850863702 [ 979.883793] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 979.890539] LustreError: 29670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 980.417475] Lustre: server umount lustre-MDT0001 complete [ 991.125331] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 993.669115] Key type lgssc unregistered [ 993.845350] LNet: 33631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 993.849613] LNetError: 33631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 993.868228] LNet: Removed LNI 192.168.202.115@tcp [ 994.779129] Key type .llcrypt unregistered [ 994.781056] Key type ._llcrypt unregistered [ 1018.191075] Key type ._llcrypt registered [ 1018.194239] Key type .llcrypt registered [ 1018.308897] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 1019.545719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1019.589092] alg: No test for adler32 (adler32-zlib) [ 1020.608904] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 1020.797679] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 1022.488815] Key type lgssc registered [ 1023.639168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1030.915800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1030.938604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1032.617069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1036.588731] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1041.891883] LustreError: 35045:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1041.922825] LustreError: 35045:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1045.736058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1046.228383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1051.698691] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1059.590651] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1066.746717] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1073.535112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1074.219039] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1081.256715] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1090.631129] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1091.419189] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1108.520654] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 12:29:39 (1787156979) [ 1111.286740] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1113.516388] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1122.893889] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 12:29:53 (1787156993) [ 1134.270867] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 12:30:05 (1787157005) [ 1135.686081] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1145.986311] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1145.992073] Lustre: Skipped 2 previous similar messages [ 1149.143099] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 12:30:19 (1787157019) [ 1152.007248] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1152.011899] Lustre: Skipped 2 previous similar messages [ 1161.036819] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1161.043668] Lustre: Skipped 4 previous similar messages [ 1179.315669] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 12:30:50 (1787157050) [ 1182.674647] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1187.178956] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1197.554099] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 12:31:08 (1787157068) [ 1199.847925] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1199.852241] Lustre: Skipped 6 previous similar messages [ 1213.795326] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 12:31:25 (1787157085) [ 1228.771054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1228.779586] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1228.780253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1228.793253] Lustre: Skipped 1 previous similar message [ 1232.080782] Lustre: server umount lustre-OST0000 complete [ 1235.937973] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1235.949131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1235.961501] Lustre: Skipped 1 previous similar message [ 1235.969558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1239.012913] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1239.014268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1239.048043] Lustre: Skipped 1 previous similar message [ 1241.671490] Lustre: server umount lustre-MDT0000 complete [ 1244.130365] LustreError: 35039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1244.157033] LustreError: 35039:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1245.030647] LustreError: 39583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787157117 with bad export cookie 7566353422379705570 [ 1245.032116] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1245.055151] LustreError: 39583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1245.383040] Lustre: server umount lustre-MDT0001 complete [ 1255.207101] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 1257.484427] Key type lgssc unregistered [ 1257.727364] LNet: 42912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1257.732328] LNetError: 42912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1258.791929] LNet: Removed LNI 192.168.202.115@tcp [ 1259.625444] Key type .llcrypt unregistered [ 1259.627301] Key type ._llcrypt unregistered [ 1277.102431] Key type ._llcrypt registered [ 1277.104869] Key type .llcrypt registered [ 1277.203188] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 1292.790165] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 1293.872633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1293.914809] alg: No test for adler32 (adler32-zlib) [ 1294.911390] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 1295.124941] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 1296.816343] Key type lgssc registered [ 1298.145923] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1333.934821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1333.965262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1335.199793] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1335.218118] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1335.321488] Lustre: lustre-MDT0000: new disk, initializing [ 1335.454124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1335.467755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1339.983469] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1350.329937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1350.438308] Lustre: 46069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1350.492297] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1350.497688] Lustre: Skipped 1 previous similar message [ 1350.560612] Lustre: lustre-MDT0001: new disk, initializing [ 1350.674117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1350.724248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1350.742216] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1355.986902] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1364.701628] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1370.575625] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1378.363105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1378.553365] Lustre: lustre-OST0000: new disk, initializing [ 1378.556165] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1378.560674] Lustre: 48036:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1378.632297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1385.684250] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1386.630907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1386.654840] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1386.739235] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1397.065220] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1403.069916] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1403.468053] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1410.553347] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1410.816386] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1417.699375] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1417.705578] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1417.714571] LustreError: Skipped 1 previous similar message [ 1417.715433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1417.750288] Lustre: Skipped 1 previous similar message [ 1419.162157] Lustre: server umount lustre-OST0000 complete [ 1427.425041] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1427.436541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1427.455073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1427.470605] Lustre: Skipped 1 previous similar message [ 1429.696710] Lustre: server umount lustre-MDT0000 complete [ 1434.480301] LustreError: 48035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787157306 with bad export cookie 7392229898693645167 [ 1434.484273] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1434.487979] LustreError: 48035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1434.743945] Lustre: server umount lustre-MDT0001 complete [ 1453.362523] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 12:35:23 (1787157323) [ 1461.134582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1461.722588] LustreError: 50505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1461.751941] LustreError: 50505:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1461.829277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1466.863683] LustreError: 50506:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1467.022552] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1471.978569] LustreError: 50505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1477.107966] LustreError: 50506:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1478.224731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1478.755884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1484.143448] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1493.751680] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1502.428573] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1511.825626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1512.316379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1519.227156] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1528.358692] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1537.806418] LustreError: 53036:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1538.600522] Lustre: Modifying parameter general.timeout=1 in log params [ 1540.351660] Lustre: Modifying parameter general.timeout=3 in log params [ 1540.358263] Lustre: Skipped 1 previous similar message [ 1543.100339] Lustre: Modifying parameter general.timeout=6 in log params [ 1543.102747] Lustre: Skipped 2 previous similar messages [ 1547.255322] Lustre: Modifying parameter general.timeout=11 in log params [ 1547.258904] Lustre: Skipped 4 previous similar messages [ 1556.270338] Lustre: Modifying parameter general.timeout=20 in log params [ 1556.275172] Lustre: Skipped 8 previous similar messages [ 1572.279612] Lustre: Modifying parameter general.timeout=35 in log params [ 1572.286557] Lustre: Skipped 14 previous similar messages [ 1604.999665] Lustre: Modifying parameter general.timeout=71 in log params [ 1605.004791] Lustre: Skipped 35 previous similar messages [ 1670.287926] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 12:39:01 (1787157541) [ 1671.479492] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1671.484438] Lustre: Skipped 59 previous similar messages [ 1673.471255] Lustre: 59979:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1675.082516] Lustre: 60083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1678.879593] Lustre: 60287:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1678.889916] Lustre: 60287:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1682.947948] Lustre: 60484:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1682.977857] Lustre: 60484:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1712.666680] Lustre: 61860:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1712.701112] LustreError: 61860:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1717.767876] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 12:39:48 (1787157588) [ 1734.632312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1734.637802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1734.649107] Lustre: Skipped 1 previous similar message [ 1734.657039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1734.659528] Lustre: Skipped 1 previous similar message [ 1736.676134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1736.677714] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1738.363489] Lustre: server umount lustre-MDT0000 complete [ 1741.697110] LustreError: 50486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787157614 with bad export cookie 7392229898693646056 [ 1741.704399] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1741.714906] LustreError: 50486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1741.798241] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1741.803606] LustreError: 50501:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1741.814264] Lustre: Skipped 1 previous similar message [ 1741.826461] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1741.834372] Lustre: Skipped 2 previous similar messages [ 1741.997660] Lustre: server umount lustre-MDT0001 complete [ 1748.355588] Lustre: server umount lustre-OST0000 complete [ 1777.571107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1777.948549] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1778.266455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1782.488213] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1791.861181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1791.993147] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1792.046962] Lustre: 64395:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1798.398618] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1808.792917] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1816.627354] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1825.041609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1825.250565] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1825.560691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1825.564907] Lustre: Skipped 1 previous similar message [ 1832.872410] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1841.445829] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1843.370813] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1843.377119] Lustre: Skipped 22 previous similar messages [ 1843.405922] Lustre: 66961:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1853.937244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1853.943334] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1853.956548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1853.958578] Lustre: Skipped 1 previous similar message [ 1853.963407] LustreError: Skipped 1 previous similar message [ 1858.252308] Lustre: server umount lustre-OST0000 complete [ 1864.168178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1868.850825] Lustre: server umount lustre-MDT0000 complete [ 1869.289504] LustreError: 65152:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1869.311603] LustreError: 65152:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1873.229670] LustreError: 67219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787157745 with bad export cookie 7392229898693649157 [ 1873.236689] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1873.240309] LustreError: 67219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1874.182821] Lustre: server umount lustre-MDT0001 complete [ 1888.604237] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 1892.661798] Key type lgssc unregistered [ 1893.079659] LNet: 68173:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1893.089557] LNetError: 68173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1893.104865] LNet: Removed LNI 192.168.202.115@tcp [ 1894.342991] Key type .llcrypt unregistered [ 1894.346853] Key type ._llcrypt unregistered [ 1925.871275] Key type ._llcrypt registered [ 1925.875995] Key type .llcrypt registered [ 1926.067332] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 1927.729540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1927.770699] alg: No test for adler32 (adler32-zlib) [ 1928.976728] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 1929.169865] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 1930.882567] Key type lgssc registered [ 1932.405425] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1940.751304] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1940.762665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1942.106173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1947.227287] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1952.226138] LustreError: 69589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1952.241642] LustreError: 69589:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1957.145936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1957.351531] LustreError: 69590:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1958.072873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1963.069640] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 1971.229339] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1978.222419] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1986.748854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1987.498986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1995.060842] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2003.247781] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2012.512131] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2023.399154] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2023.405376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2023.417504] Lustre: Skipped 1 previous similar message [ 2023.427624] Lustre: Skipped 2 previous similar messages [ 2028.515190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2028.532102] Lustre: Skipped 2 previous similar messages [ 2028.934933] Lustre: server umount lustre-MDT0000 complete [ 2033.319480] LustreError: 71441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787157905 with bad export cookie 18013114654047108768 [ 2033.328815] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2033.330209] LustreError: 71441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2033.595101] Lustre: server umount lustre-MDT0001 complete [ 2047.905296] Lustre: server umount lustre-OST0000 complete [ 2081.138951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2081.526556] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2081.555420] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2081.909629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2086.737338] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2096.493459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2096.587516] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2096.624851] Lustre: 74461:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2097.068085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2102.002149] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2110.335244] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2117.163660] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2124.871410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2125.218892] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2125.699270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2133.647815] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2143.021904] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2146.393537] Lustre: 77040:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2148.662384] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2157.543121] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2157.555234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2157.569398] Lustre: Skipped 1 previous similar message [ 2157.577724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2158.049734] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2161.463808] Lustre: server umount lustre-OST0000 complete [ 2168.297102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2168.303760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2168.331952] Lustre: Skipped 2 previous similar messages [ 2168.350403] Lustre: Skipped 3 previous similar messages [ 2173.417968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2173.434135] Lustre: Skipped 1 previous similar message [ 2180.065420] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2182.346980] Lustre: server umount lustre-MDT0000 complete [ 2182.634602] LustreError: 74474:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2182.644150] LustreError: 74474:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2186.771398] LustreError: 74453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787158059 with bad export cookie 18013114654047109615 [ 2186.774919] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2186.779580] LustreError: 74453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2187.365147] Lustre: server umount lustre-MDT0001 complete [ 2201.472414] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 2204.443450] Key type lgssc unregistered [ 2204.942571] LNet: 78251:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2204.948737] LNetError: 78251:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2204.963828] LNet: Removed LNI 192.168.202.115@tcp [ 2205.837136] Key type .llcrypt unregistered [ 2205.839262] Key type ._llcrypt unregistered [ 2235.657527] Key type ._llcrypt registered [ 2235.660235] Key type .llcrypt registered [ 2235.743784] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 2238.156451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2238.190017] alg: No test for adler32 (adler32-zlib) [ 2239.274526] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 2239.602464] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 2241.360201] Key type lgssc registered [ 2242.763546] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2251.767595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2251.802104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2253.624896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2259.757762] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2263.012600] LustreError: 79666:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2263.057574] LustreError: 79666:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2268.130165] LustreError: 79665:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2270.730132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2271.514784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2275.971863] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2284.778643] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2292.256365] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2301.087333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2301.798175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2309.655049] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2318.929901] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2330.608636] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2331.817904] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2332.981540] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2335.364128] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2335.368630] Lustre: Skipped 1 previous similar message [ 2340.094300] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2340.101119] Lustre: Skipped 4 previous similar messages [ 2348.290154] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2348.295366] Lustre: Skipped 8 previous similar messages [ 2364.666852] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2364.682891] Lustre: Skipped 15 previous similar messages [ 2397.557925] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2397.561056] Lustre: Skipped 36 previous similar messages [ 2431.698487] Lustre: 87767:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2435.224619] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 12:51:46 (1787158306) [ 2437.032398] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2439.103323] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 12:51:50 (1787158310) [ 2444.257941] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2444.274099] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2444.292100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2446.306159] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2446.308682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2446.323980] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2446.353020] Lustre: Skipped 1 previous similar message [ 2450.023835] Lustre: server umount lustre-OST0000 complete [ 2455.523409] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2455.539141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2455.560117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2460.518563] Lustre: server umount lustre-MDT0000 complete [ 2461.684869] LustreError: 79665:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2461.709825] LustreError: 79665:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2464.928397] LustreError: 79646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787158337 with bad export cookie 2506793628920189558 [ 2464.942592] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2464.947973] LustreError: 79646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2465.348813] Lustre: server umount lustre-MDT0001 complete [ 2480.444352] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 2484.024510] Key type lgssc unregistered [ 2484.504663] LNet: 89022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2484.523217] LNetError: 89022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2484.541534] LNet: Removed LNI 192.168.202.115@tcp [ 2485.607892] Key type .llcrypt unregistered [ 2485.610852] Key type ._llcrypt unregistered [ 2490.100422] Key type ._llcrypt registered [ 2490.104550] Key type .llcrypt registered [ 2490.207664] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2494.710836] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules [ 2496.067415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2496.348888] alg: No test for adler32 (adler32-zlib) [ 2497.407176] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 2497.609485] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 2499.352185] Key type lgssc registered [ 2500.619622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2502.014377] LustreError: 90224:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2522.136856] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 2524.485363] LustreError: 90224:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2531.662494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2531.677330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2533.359244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2538.573178] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2543.589190] LustreError: 91248:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2543.609449] LustreError: 91248:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2547.916315] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 12:53:39 (1787158419) [ 2548.710711] LustreError: 91249:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2552.141424] LustreError: 91280:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2552.155817] LustreError: 91280:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2552.164411] LustreError: 91280:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 2552.493903] Lustre: server umount lustre-MDT0000 complete [ 2565.974383] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 2569.328736] Key type lgssc unregistered [ 2569.644190] LNet: 92771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2569.650819] LNetError: 92771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2570.669967] LNet: Removed LNI 192.168.202.115@tcp [ 2571.568741] Key type .llcrypt unregistered [ 2571.571260] Key type ._llcrypt unregistered [ 2591.173415] Key type ._llcrypt registered [ 2591.175285] Key type .llcrypt registered [ 2591.255939] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 2592.215772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2592.350031] alg: No test for adler32 (adler32-zlib) [ 2593.352150] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 2593.599207] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 2595.272214] Key type lgssc registered [ 2596.221337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2605.036518] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2605.048753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2606.490789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2611.229222] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2615.784840] LustreError: 93992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2615.805572] LustreError: 93992:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2619.722701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2620.352894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2625.153102] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2633.395692] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2638.824452] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2644.997706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2645.629213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2651.244703] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2657.808337] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2711.804460] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 12:56:23 (1787158583) [ 2713.618722] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2715.568833] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 12:56:27 (1787158587) [ 2721.762358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2721.766606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2721.783508] Lustre: Skipped 1 previous similar message [ 2721.798977] Lustre: Skipped 2 previous similar messages [ 2725.439392] Lustre: server umount lustre-MDT0000 complete [ 2726.885566] LustreError: 93992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2726.906836] LustreError: 93992:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2729.559118] LustreError: 97168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787158601 with bad export cookie 17673884030876153843 [ 2729.560882] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2729.582948] LustreError: 97168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2729.807902] Lustre: server umount lustre-MDT0001 complete [ 2744.418979] Lustre: server umount lustre-OST0000 complete [ 2757.498427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2757.882295] LustreError: 98288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2758.026544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2762.840828] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2763.241110] LustreError: 98289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2763.259623] LustreError: 98289:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2768.356484] LustreError: 98288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2771.071780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2771.972933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2775.951498] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2783.902513] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2790.664887] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2805.902536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2806.092266] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2806.103149] LustreError: 98278:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2806.153555] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2806.160742] LustreError: 100370:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2806.163366] LustreError: 100370:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2806.179720] LustreError: 100370:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2806.247751] Lustre: server umount lustre-OST0000 complete [ 2806.250568] LustreError: 100370:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2816.944236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2817.077126] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2817.081946] LustreError: 98276:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2817.112812] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2817.118587] LustreError: 100841:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2817.123715] LustreError: 100841:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2817.126658] LustreError: 100841:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2817.183457] Lustre: server umount lustre-OST0000 complete [ 2817.186213] LustreError: 100841:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2828.211366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2828.326597] LustreError: 101359:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2828.436825] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2828.440795] Lustre: Found index 0 for lustre-OST0000, updating log [ 2828.447698] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2828.557765] Lustre: lustre-OST0000: new disk, initializing [ 2828.560596] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2828.567978] Lustre: 101376:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2828.858176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2835.595949] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2838.995949] Lustre: Failing over lustre-OST0000 [ 2839.017519] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2839.021595] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2839.036497] Lustre: Skipped 1 previous similar message [ 2839.043668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2839.164679] Lustre: server umount lustre-OST0000 complete [ 2846.178467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2846.182340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2846.192156] Lustre: Skipped 1 previous similar message [ 2849.465651] Lustre: server umount lustre-MDT0000 complete [ 2851.300891] LustreError: 98285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2851.327062] LustreError: 98285:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2852.698186] LustreError: 100392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787158725 with bad export cookie 17673884030876156587 [ 2852.700592] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2852.708457] LustreError: 100392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2852.989341] Lustre: server umount lustre-MDT0001 complete [ 2860.813364] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 12:58:52 (1787158732) [ 2877.308647] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 2887.709869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2887.990565] LustreError: 103721:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2888.090248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2892.534871] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2900.868238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2905.616735] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2908.304383] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2914.961152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2915.487376] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2922.229477] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2925.732920] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2925.739085] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2925.773486] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2930.250606] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2930.366096] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2930.480654] Lustre: lustre-OST0001: new disk, initializing [ 2930.489770] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2930.501692] Lustre: 106039:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2930.794095] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2930.813783] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2930.830894] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2930.834157] Lustre: Skipped 2 previous similar messages [ 2932.905648] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:02 [ 2932.922270] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2932.926528] Lustre: lustre-OST0000: Recovery over after 0:07, of 1 clients 1 recovered and 0 were evicted. [ 2932.932649] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2932.943859] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2937.377697] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2937.490112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2937.499780] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2937.643339] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2945.271342] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2955.286344] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2960.780445] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.15@tcp (stopping) [ 2960.787525] Lustre: Skipped 1 previous similar message [ 2963.427343] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2963.442428] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2963.446118] Lustre: Skipped 1 previous similar message [ 2965.892936] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.15@tcp (stopping) [ 2965.909803] Lustre: Skipped 3 previous similar messages [ 2966.701324] Lustre: server umount lustre-MDT0001 complete [ 2968.567500] LustreError: 103721:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2968.586040] LustreError: 103721:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2976.030390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2976.099198] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2976.101845] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2976.140819] Lustre: 103708:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2977.984594] LustreError: 107616:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2978.001311] LustreError: 107616:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2978.019076] LustreError: 103713:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2978.130083] LustreError: 107625:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.115@tcp: cfg command failed: rc = -17 [ 2978.135804] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2978.135804] [ 2981.079717] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 2981.874476] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2991.572370] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 13:01:03 (1787158863) [ 3007.949443] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 3028.474583] Lustre: 109664:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3054.145680] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3059.110105] Lustre: 110801:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3145.184957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3145.200323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3145.211938] Lustre: Skipped 1 previous similar message [ 3145.223928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3150.285126] Lustre: server umount lustre-MDT0000 complete [ 3150.819939] LustreError: 103717:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3150.831748] LustreError: 103717:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 3153.190391] LustreError: 106038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787159025 with bad export cookie 17673884030876157581 [ 3153.204394] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3153.213119] LustreError: 106038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3153.467658] Lustre: server umount lustre-MDT0001 complete [ 3167.473233] Lustre: server umount lustre-OST0000 complete [ 3181.329869] Lustre: server umount lustre-OST0001 complete [ 3186.879430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3196.406367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3204.676419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3213.335807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3231.350673] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 3240.510767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3240.527548] Lustre: lustre-MDT0000: reset Object Index mappings [ 3240.894870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3240.899506] Lustre: Skipped 1 previous similar message [ 3244.636236] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 3251.566793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3251.590179] Lustre: lustre-MDT0001: reset Object Index mappings [ 3256.141755] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 3259.417429] Lustre: 115901:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3266.153469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3272.197969] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 3273.499914] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 3277.312905] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3283.154379] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3289.117142] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3289.121188] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3290.925047] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 3299.102472] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3303.195668] Lustre: 117776:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3350.910705] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 13:07:02 (1787159222) [ 3380.197946] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3380.206727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3380.232348] Lustre: Skipped 3 previous similar messages [ 3380.248638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3380.259990] Lustre: Skipped 3 previous similar messages [ 3384.921316] Lustre: server umount lustre-MDT0000 complete [ 3388.710548] LustreError: 114735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787159261 with bad export cookie 17673884030876204705 [ 3388.728960] LustreError: 114735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3388.737390] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3403.232931] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3403.574749] Lustre: server umount lustre-MDT0001 complete [ 3413.847740] Lustre: server umount lustre-OST0000 complete [ 3418.010773] LustreError: 118738:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0001_UUID: namespace resource [0x2c0000401:0x3a:0x0].0x0 (ffff9a10c3156f00) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 3424.362202] Lustre: server umount lustre-OST0001 complete [ 3432.073977] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 3444.943996] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 3485.788269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3486.237121] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3486.322188] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3486.557708] Lustre: lustre-MDT0000: new disk, initializing [ 3486.798395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3486.819028] Lustre: Skipped 3 previous similar messages [ 3486.903450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3495.203341] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 3503.418213] Lustre: server umount lustre-MDT0000 complete [ 3514.501795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3514.945099] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3514.970935] Lustre: 122046:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3514.998947] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3519.618692] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 3525.025610] Lustre: server umount lustre-MDT0000 complete [ 3535.542585] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 13:10:06 (1787159406) [ 3537.742664] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3539.683990] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 13:10:11 (1787159411) [ 3560.517581] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 3570.955372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3575.756736] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 3585.488672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3585.710452] Lustre: 124097:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3585.802954] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3585.811864] Lustre: Skipped 1 previous similar message [ 3585.961978] Lustre: lustre-MDT0001: new disk, initializing [ 3586.150797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3586.172814] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3592.082657] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 3598.463535] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3606.974733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3607.252966] Lustre: lustre-OST0000: new disk, initializing [ 3607.257315] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3607.264194] Lustre: 125677:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3608.496451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3608.513081] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3608.649901] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3614.566160] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 3627.572677] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3627.777990] Lustre: lustre-OST0001: new disk, initializing [ 3627.783826] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3627.791439] Lustre: 126547:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3628.966493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3628.974686] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3629.097224] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3634.222565] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 3646.312522] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3651.949429] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4091.457600] Lustre: 132972:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4091.468063] Lustre: 132972:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4091.473079] Lustre: 132972:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4091.477441] Lustre: 132972:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4091.486388] Lustre: 132972:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4091.495543] Lustre: 132972:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4366.618820] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 13:23:57 (1787160237) [ 4384.737895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4384.747945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4384.769496] Lustre: Skipped 5 previous similar messages [ 4384.778582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4384.787830] Lustre: Skipped 10 previous similar messages [ 4386.794106] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4386.821728] Lustre: Skipped 2 previous similar messages [ 4390.145064] Lustre: server umount lustre-MDT0000 complete [ 4391.924281] LustreError: 124108:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4391.952159] LustreError: 124108:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 23 previous similar messages [ 4394.777235] LustreError: 125676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787160267 with bad export cookie 17673884030876271856 [ 4394.789520] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4394.792981] LustreError: 125676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4397.028499] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4397.035511] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4397.053274] Lustre: Skipped 1 previous similar message [ 4397.061981] Lustre: Skipped 4 previous similar messages [ 4405.228656] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4405.234558] Lustre: Skipped 4 previous similar messages [ 4409.312492] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4409.810272] Lustre: server umount lustre-MDT0001 complete [ 4419.890623] Lustre: server umount lustre-OST0000 complete [ 4429.511937] Lustre: server umount lustre-OST0001 complete [ 4437.407867] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 4448.816563] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 4484.925918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4485.189979] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4485.226126] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4485.283740] Lustre: lustre-MDT0000: new disk, initializing [ 4485.339093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4485.344570] Lustre: Skipped 5 previous similar messages [ 4485.353658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4489.545829] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 4499.790282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4499.922753] Lustre: 136023:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4499.988590] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4499.991129] Lustre: Skipped 1 previous similar message [ 4500.094531] Lustre: lustre-MDT0001: new disk, initializing [ 4500.285746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4500.300399] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4505.046953] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 4514.025269] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4520.216154] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4526.906611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4527.128652] Lustre: lustre-OST0000: new disk, initializing [ 4527.133161] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4527.144246] Lustre: 137991:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4527.244750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4527.250862] Lustre: Skipped 1 previous similar message [ 4528.716884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4528.728452] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4528.871697] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4533.692581] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 4543.862841] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4553.093190] Lustre: lustre-MDD0000: changelog on [ 4555.920166] Lustre: lustre-MDD0001: changelog on [ 6375.740629] Lustre: lustre-MDD0001: changelog off [ 6377.972651] Lustre: 136905:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6379.114960] Lustre: lustre-MDD0000: changelog off [ 6392.804346] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6392.823860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6392.824298] Lustre: Skipped 1 previous similar message [ 6392.826713] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6392.831132] Lustre: Skipped 2 previous similar messages [ 6394.375791] Lustre: server umount lustre-OST0000 complete [ 6399.972586] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6399.984062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6400.012932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6403.041625] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6405.018707] Lustre: server umount lustre-MDT0000 complete [ 6408.163342] LustreError: 136033:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6408.174030] LustreError: 136033:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 6409.374181] LustreError: 136815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787162281 with bad export cookie 17673884030877095609 [ 6409.386362] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6409.392369] LustreError: 136815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6409.760448] Lustre: server umount lustre-MDT0001 complete [ 6422.804204] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 6425.619876] Key type lgssc unregistered [ 6425.967792] LNet: 147121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6425.975641] LNetError: 147121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6425.993748] LNet: Removed LNI 192.168.202.115@tcp [ 6427.035159] Key type .llcrypt unregistered [ 6427.043212] Key type ._llcrypt unregistered [ 6446.466246] Key type ._llcrypt registered [ 6446.469272] Key type .llcrypt registered [ 6446.582519] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 6464.582866] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 6465.844526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6465.980182] alg: No test for adler32 (adler32-zlib) [ 6467.145717] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 6467.489243] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 6469.232500] Key type lgssc registered [ 6470.665842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6511.155952] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6511.182141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6512.433766] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6512.472768] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6512.568269] Lustre: lustre-MDT0000: new disk, initializing [ 6512.730272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6512.753403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6517.097065] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6528.458440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6528.567991] Lustre: 150325:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6528.606262] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6528.606885] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6528.615905] Lustre: Skipped 1 previous similar message [ 6528.666501] Lustre: lustre-MDT0001: new disk, initializing [ 6528.797053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6528.853018] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6528.867275] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6533.158485] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6543.301181] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6550.290639] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6558.075846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6558.451567] Lustre: lustre-OST0000: new disk, initializing [ 6558.460819] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6558.472276] Lustre: 152297:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6558.608821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6565.948498] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6568.505126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6568.515366] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6568.604626] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6577.095552] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6584.252491] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_module obdecho/obdecho [ 6586.826117] LustreError: 153301:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6586.840420] LustreError: 153301:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6589.343853] LustreError: 153397:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6604.770677] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6604.775924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6604.802934] Lustre: Skipped 2 previous similar messages [ 6604.808310] Lustre: Skipped 1 previous similar message [ 6608.810523] Lustre: server umount lustre-MDT0000 complete [ 6613.498032] LustreError: 150319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787162485 with bad export cookie 8381760343376083281 [ 6613.507977] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6613.522367] LustreError: 150319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6613.863156] Lustre: server umount lustre-MDT0001 complete [ 6629.831511] Lustre: server umount lustre-OST0000 complete [ 6652.547222] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 6666.588083] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 6715.081841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6715.331559] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6715.367586] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6715.448435] Lustre: lustre-MDT0000: new disk, initializing [ 6715.516353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6715.528337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6719.828413] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6732.980792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6733.191735] Lustre: 157130:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6733.252432] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6733.256336] Lustre: Skipped 1 previous similar message [ 6733.476605] Lustre: lustre-MDT0001: new disk, initializing [ 6733.621345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6733.668052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6733.688651] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6739.675880] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6751.548597] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6758.162582] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6766.260645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6766.528982] Lustre: lustre-OST0000: new disk, initializing [ 6766.532403] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6766.537737] Lustre: 159096:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6766.625350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6772.818457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6772.827990] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6772.957461] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6774.273375] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6785.445315] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6790.820959] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6791.082553] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6796.141517] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6796.417336] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6798.818398] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6798.825053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6798.826114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6798.831354] Lustre: Skipped 1 previous similar message [ 6798.840067] Lustre: Skipped 2 previous similar messages [ 6803.941123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6803.952033] Lustre: Skipped 1 previous similar message [ 6804.658263] Lustre: server umount lustre-OST0000 complete [ 6809.059828] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6809.063633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6809.097302] Lustre: Skipped 1 previous similar message [ 6809.115204] Lustre: Skipped 1 previous similar message [ 6815.003488] Lustre: server umount lustre-MDT0000 complete [ 6818.452967] LustreError: 157120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787162690 with bad export cookie 8381760343376084240 [ 6818.456797] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6818.462517] LustreError: 157120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6818.716508] Lustre: server umount lustre-MDT0001 complete [ 6822.510861] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 14:04:53 (1787162693) [ 6824.018355] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6826.309838] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 14:04:57 (1787162697) [ 6835.016284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6835.525145] LustreError: 161511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6835.553490] LustreError: 161511:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6835.641456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6840.814207] LustreError: 161512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6840.844645] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6844.963701] LustreError: 161511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6850.023702] LustreError: 161512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6851.858259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6858.236679] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6866.936375] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6872.724761] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6880.216644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6880.665110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6880.675782] Lustre: Skipped 1 previous similar message [ 6886.994333] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 6894.373164] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6919.137417] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6919.141021] LustreError: Skipped 1 previous similar message [ 6919.157311] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6919.179875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6919.182726] Lustre: Skipped 2 previous similar messages [ 6922.168179] Lustre: server umount lustre-MDT0000 complete [ 6925.679461] LustreError: 164016:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6926.746728] LustreError: 161490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787162799 with bad export cookie 8381760343376085122 [ 6926.751607] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6926.768548] LustreError: 161490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6930.916308] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6930.934296] Lustre: Skipped 2 previous similar messages [ 6935.917613] LustreError: 161506:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6935.933900] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.15@tcp (stopping) [ 6935.951564] LustreError: 161506:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 6935.969330] Lustre: Skipped 7 previous similar messages [ 6941.152232] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6941.687375] Lustre: server umount lustre-MDT0001 complete [ 6947.710101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6962.914841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6963.439690] LustreError: 165378:0:(llog.c:1655:llog_backup()) MGC192.168.202.115@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6963.453407] Lustre: 165378:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.115@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6966.756485] LustreError: 165384:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6966.795541] LustreError: 165384:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7001.470829] LustreError: 165384:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7001.518037] LustreError: 165384:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 25 previous similar messages [ 7024.291024] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7024.298742] Lustre: 165378:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.115@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7065.089955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7066.999036] LustreError: 165383:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7067.024610] LustreError: 165383:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 44 previous similar messages [ 7070.100967] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7070.196781] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7079.903977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7092.716755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7100.759796] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7102.438894] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7109.145935] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7111.057779] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7117.290649] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7119.148049] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7141.251556] Lustre: server umount lustre-MDT0000 complete [ 7143.400453] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7143.404435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7145.215420] LustreError: 161492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787163017 with bad export cookie 8381760343376091611 [ 7145.218827] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7145.224514] LustreError: 161492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7145.981593] Lustre: server umount lustre-MDT0001 complete [ 7160.837850] Lustre: server umount lustre-OST0000 complete [ 7186.188745] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 7198.883561] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 7236.889091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7237.278844] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7237.306586] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7237.380237] Lustre: lustre-MDT0000: new disk, initializing [ 7237.463581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7237.470944] Lustre: Skipped 1 previous similar message [ 7237.493649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7243.376873] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7256.352172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7256.506567] Lustre: 170491:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7256.795456] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7263.426660] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7276.582933] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7286.139166] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7292.714580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7292.952367] Lustre: lustre-OST0000: new disk, initializing [ 7292.955782] Lustre: Skipped 1 previous similar message [ 7292.958817] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7292.964046] Lustre: Skipped 2 previous similar messages [ 7292.968531] Lustre: 172459:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7301.558853] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7302.211986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7302.223358] Lustre: Skipped 1 previous similar message [ 7302.231628] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7302.391621] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7313.288493] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7318.454876] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7318.640402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7324.417926] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7324.640910] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7328.227668] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7328.228690] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7328.237212] LustreError: Skipped 1 previous similar message [ 7328.241553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7328.241561] Lustre: Skipped 2 previous similar messages [ 7328.294420] Lustre: Skipped 3 previous similar messages [ 7332.827142] Lustre: server umount lustre-OST0000 complete [ 7338.468774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7342.414719] Lustre: server umount lustre-MDT0000 complete [ 7343.585734] LustreError: 170503:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7343.599705] LustreError: 170503:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 31 previous similar messages [ 7345.995039] LustreError: 170484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787163218 with bad export cookie 8381760343376108418 [ 7346.004048] LustreError: 170484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7346.004087] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7346.375036] Lustre: server umount lustre-MDT0001 complete [ 7356.743979] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 14:13:47 (1787163227) [ 7362.634901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7367.738956] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7375.575335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7381.117415] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7389.844300] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7396.676780] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7403.579495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7410.899730] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7419.170148] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7435.690827] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 14:15:06 (1787163306) [ 7441.888955] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7441.889939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7441.919715] Lustre: Skipped 2 previous similar messages [ 7441.928628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7441.934447] Lustre: Skipped 4 previous similar messages [ 7445.762867] Lustre: server umount lustre-OST0000 complete [ 7457.022837] Lustre: server umount lustre-MDT0000 complete [ 7461.781746] LustreError: 174853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787163334 with bad export cookie 8381760343376109293 [ 7461.788677] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7461.795468] LustreError: 174853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7462.594850] Lustre: server umount lustre-MDT0001 complete [ 7476.947612] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 7480.295766] Key type lgssc unregistered [ 7480.536645] LNet: 178601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7480.555613] LNetError: 178601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7480.589484] LNet: Removed LNI 192.168.202.115@tcp [ 7481.482704] Key type .llcrypt unregistered [ 7481.485208] Key type ._llcrypt unregistered [ 7503.282752] Key type ._llcrypt registered [ 7503.286698] Key type .llcrypt registered [ 7503.445756] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 7504.910924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7505.082631] alg: No test for adler32 (adler32-zlib) [ 7506.098972] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 7506.440172] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 7508.163056] Key type lgssc registered [ 7509.678238] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7517.327935] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7517.347896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7532.896529] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7538.016674] LustreError: 179845:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.115@tcp: failed processing log, type 4: rc = -110 [ 7543.137568] LustreError: 179845:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7543.156079] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7543.164295] LustreError: MGC192.168.202.115@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7543.180634] LustreError: 179845:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7543.197661] LustreError: 179845:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7543.202578] LustreError: 179845:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7543.271874] Lustre: server umount lustre-OST0000 complete [ 7543.281527] LustreError: 179845:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7552.898068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7554.450164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7559.256346] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7563.748669] LustreError: 180345:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7563.767230] LustreError: 180345:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7568.020323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7568.359652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7573.416870] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7583.432351] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7593.626244] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7601.920190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7602.276446] LustreError: 182161:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7602.286271] Lustre: 182161:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.115@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7602.468916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7611.157345] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7620.651531] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7624.162642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7624.178920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7624.193724] Lustre: Skipped 2 previous similar messages [ 7624.208949] Lustre: Skipped 2 previous similar messages [ 7629.298237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7629.309589] Lustre: Skipped 2 previous similar messages [ 7629.576841] Lustre: server umount lustre-MDT0000 complete [ 7634.034863] LustreError: 180327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787163506 with bad export cookie 11244393876052019717 [ 7634.046402] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7634.055946] LustreError: 180327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7634.397076] Lustre: server umount lustre-MDT0001 complete [ 7649.112805] Lustre: server umount lustre-OST0000 complete [ 7658.660754] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 7671.707196] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 7714.160890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7714.382318] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7714.406339] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7714.463209] Lustre: lustre-MDT0000: new disk, initializing [ 7714.541676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7714.559669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7719.590904] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7731.573347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7731.724861] Lustre: 185689:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7731.770499] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7731.774133] Lustre: Skipped 1 previous similar message [ 7731.831291] Lustre: lustre-MDT0001: new disk, initializing [ 7731.919834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7731.954304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7731.975598] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7736.993443] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7748.195137] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7755.295697] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7763.524654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7763.829517] Lustre: lustre-OST0000: new disk, initializing [ 7763.832587] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7763.839702] Lustre: 187657:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7763.947715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7770.479855] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7772.780847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7772.790957] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7772.904640] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7781.628281] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7788.026966] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7788.419206] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7794.995994] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7795.483380] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7798.756267] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7798.759810] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7798.769650] LustreError: Skipped 1 previous similar message [ 7798.775031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7798.784051] Lustre: Skipped 1 previous similar message [ 7798.820991] Lustre: Skipped 1 previous similar message [ 7803.884133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7803.899598] Lustre: Skipped 1 previous similar message [ 7804.679928] Lustre: server umount lustre-OST0000 complete [ 7814.119726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7814.121895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7814.123649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7814.145542] Lustre: Skipped 1 previous similar message [ 7815.713573] Lustre: server umount lustre-MDT0000 complete [ 7819.238940] LustreError: 185702:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7819.279444] LustreError: 185702:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7820.189501] LustreError: 185684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787163692 with bad export cookie 11244393876052020431 [ 7820.195891] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7820.203496] LustreError: 185684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7821.136762] Lustre: server umount lustre-MDT0001 complete [ 7836.197761] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 14:21:46 (1787163706) [ 7859.859717] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 7862.769931] Key type lgssc unregistered [ 7863.245069] LNet: 190610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7863.253863] LNetError: 190610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7863.285960] LNet: Removed LNI 192.168.202.115@tcp [ 7864.403551] Key type .llcrypt unregistered [ 7864.409721] Key type ._llcrypt unregistered [ 7888.367081] Key type ._llcrypt registered [ 7888.368771] Key type .llcrypt registered [ 7888.503877] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 7889.992910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7890.075523] alg: No test for adler32 (adler32-zlib) [ 7891.136662] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 7891.377840] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 7893.050489] Key type lgssc registered [ 7894.303588] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7902.380040] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7902.410063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7903.919894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7909.222711] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7914.475208] LustreError: 191829:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7914.503585] LustreError: 191829:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7915.494197] LustreError: 191830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7919.227711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7919.871879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7925.545600] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7934.922089] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7943.162655] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7952.583392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7952.903744] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7952.908477] LustreError: 193693:0:(llog.c:1696:llog_backup()) MGC192.168.202.115@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7952.919560] Lustre: 193693:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.115@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7953.055792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7960.643565] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 7969.682907] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7975.925736] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7975.949490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7975.966867] Lustre: Skipped 1 previous similar message [ 7975.970665] Lustre: Skipped 2 previous similar messages [ 7981.046680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7981.066216] Lustre: Skipped 2 previous similar messages [ 7986.162825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7986.188752] Lustre: Skipped 3 previous similar messages [ 7987.170781] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7987.480520] Lustre: server umount lustre-MDT0000 complete [ 7991.268097] LustreError: 193118:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7991.298779] LustreError: 193118:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7992.076152] LustreError: 191810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787163864 with bad export cookie 8638653419389762244 [ 7992.088408] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7992.096752] LustreError: 191810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7992.753259] Lustre: server umount lustre-MDT0001 complete [ 8009.349638] Lustre: server umount lustre-OST0000 complete [ 8021.066091] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 8037.355996] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 8085.148055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8085.570859] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8085.630780] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8085.736689] Lustre: lustre-MDT0000: new disk, initializing [ 8085.982995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8086.030789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8093.618691] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8107.359746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8107.483954] Lustre: 197243:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8107.523968] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8107.526246] Lustre: Skipped 1 previous similar message [ 8107.529598] LustreError: 198028:0:(llog.c:1696:llog_backup()) MGC192.168.202.115@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8107.534721] LustreError: 198028:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8107.538911] Lustre: 198028:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.115@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8107.555350] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8107.558656] Lustre: Skipped 1 previous similar message [ 8107.639520] Lustre: lustre-MDT0001: new disk, initializing [ 8107.743846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8107.788984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8107.801174] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8113.599640] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8124.958455] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8132.032567] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8139.933303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8140.133633] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8140.136218] Lustre: Skipped 1 previous similar message [ 8140.143486] LustreError: 199190:0:(llog.c:1696:llog_backup()) MGC192.168.202.115@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8140.151126] LustreError: 199190:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8140.156658] Lustre: 199190:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.115@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8140.163732] Lustre: 199190:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8140.178637] Lustre: lustre-OST0000: new disk, initializing [ 8140.181651] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8140.185248] Lustre: 199213:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8140.270164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8148.105030] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8149.114808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8149.143828] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8149.294386] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8160.911164] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8167.865662] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8168.211459] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8174.056253] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8174.564841] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8180.195101] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8180.198496] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8180.239824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8182.962622] Lustre: server umount lustre-OST0000 complete [ 8189.922873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8189.946835] LustreError: Skipped 1 previous similar message [ 8189.953630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8189.969884] Lustre: Skipped 1 previous similar message [ 8189.974572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8189.978379] Lustre: Skipped 1 previous similar message [ 8199.659865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8199.677934] Lustre: Skipped 5 previous similar messages [ 8203.232529] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8203.858041] Lustre: server umount lustre-MDT0000 complete [ 8204.773436] LustreError: 198375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8209.201767] LustreError: 198040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787164081 with bad export cookie 8638653419389762958 [ 8209.204071] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8209.213728] LustreError: 198040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8209.446365] Lustre: server umount lustre-MDT0001 complete [ 8223.785256] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 14:28:13 (1787164093) [ 8261.196340] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 8279.201304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8279.626156] LustreError: 202370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8279.650391] LustreError: 202370:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8279.685493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8286.356392] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8290.286126] LustreError: 202371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8290.316673] LustreError: 202371:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8299.405269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8306.533514] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8310.283319] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8318.172499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8318.426308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8318.429437] Lustre: Skipped 1 previous similar message [ 8326.698878] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8338.173204] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8338.383149] Lustre: lustre-OST0001: new disk, initializing [ 8338.388192] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8338.396462] Lustre: 204679:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8346.281365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8346.298337] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8346.468499] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8347.039182] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8358.010884] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8365.218627] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8367.544922] Lustre: DEBUG MARKER: ADD OST3 [ 8374.193767] loop: module loaded [ 8374.308547] loop0: detected capacity change from 0 to 204800000 [ 8374.676950] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8374.685462] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8374.694272] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8383.639207] Lustre: DEBUG MARKER: STOP OST3 [ 8384.544389] loop0: detected capacity change from 0 to 204800000 [ 8393.341051] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8393.672501] Lustre: lustre-OST0003: new disk, initializing [ 8393.676349] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8393.684426] Lustre: 206765:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8393.857166] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8393.870077] Lustre: Skipped 1 previous similar message [ 8395.183887] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8395.481460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8395.495368] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8395.566117] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8396.423445] Lustre: Failing over lustre-OST0003 [ 8396.903147] Lustre: server umount lustre-OST0003 complete [ 8397.222744] LustreError: 206849:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8400.322862] LustreError: 204165:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8400.347263] LustreError: 204165:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8400.395168] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8400.441955] Lustre: Skipped 2 previous similar messages [ 8403.542426] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8417.828890] Lustre: DEBUG MARKER: START OST3 again [ 8427.096607] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8427.396904] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8428.973604] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8429.212433] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8429.224368] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8437.632720] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8462.512651] Lustre: Failing over lustre-OST0003 [ 8462.709075] Lustre: server umount lustre-OST0003 complete [ 8463.355498] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8463.366351] LustreError: 203856:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8463.382794] Lustre: Skipped 1 previous similar message [ 8463.415770] LustreError: 203856:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 8483.827718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8483.831692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8483.839472] Lustre: Skipped 2 previous similar messages [ 8483.857331] Lustre: Skipped 1 previous similar message [ 8497.120576] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8497.469118] Lustre: server umount lustre-MDT0000 complete [ 8501.603111] LustreError: 204678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787164374 with bad export cookie 8638653419389763840 [ 8501.604307] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8501.608982] LustreError: 204678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8504.309091] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8504.329020] Lustre: Skipped 1 previous similar message [ 8507.968875] Lustre: server umount lustre-MDT0001 complete [ 8512.659350] Lustre: server umount lustre-OST0000 complete [ 8517.244442] Lustre: server umount lustre-OST0001 complete [ 8524.730161] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 8535.641965] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 8583.311720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8583.870104] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8583.937882] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8584.075188] Lustre: lustre-MDT0000: new disk, initializing [ 8584.193336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8584.208302] Lustre: Skipped 1 previous similar message [ 8584.238442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8590.149771] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8603.581041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8603.746906] Lustre: 212296:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8603.795541] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8603.798716] Lustre: Skipped 1 previous similar message [ 8603.851188] Lustre: lustre-MDT0001: new disk, initializing [ 8603.978188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8603.987359] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8608.166245] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8620.649921] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8628.069621] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8637.329885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8637.646652] Lustre: lustre-OST0000: new disk, initializing [ 8637.651617] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8637.657411] Lustre: 214264:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8639.874119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8639.894041] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8639.979791] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8644.906236] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8654.974564] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8660.591669] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8660.974094] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8665.654863] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8665.905393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8670.691176] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8670.694550] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8670.715208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8670.715217] Lustre: Skipped 12 previous similar messages [ 8670.750337] Lustre: Skipped 1 previous similar message [ 8674.247332] Lustre: server umount lustre-OST0000 complete [ 8680.930547] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8680.948173] LustreError: Skipped 1 previous similar message [ 8693.216204] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8693.482840] Lustre: server umount lustre-MDT0000 complete [ 8696.289542] LustreError: 212301:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8696.317161] LustreError: 212301:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 20 previous similar messages [ 8698.439137] LustreError: 212287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787164570 with bad export cookie 8638653419389766136 [ 8698.441325] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8698.446919] LustreError: 212287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8698.879807] Lustre: server umount lustre-MDT0001 complete [ 8703.535668] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 14:36:14 (1787164574) [ 8723.861977] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 8742.924861] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 8791.504974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8791.857751] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8791.909494] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8792.034855] Lustre: lustre-MDT0000: new disk, initializing [ 8792.191404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8798.013692] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8811.345286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8811.498870] Lustre: 218901:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8811.877681] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8817.710544] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8831.488402] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8839.709453] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8851.440288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8852.318330] Lustre: 220871:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8852.594218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8852.610503] Lustre: Skipped 4 previous similar messages [ 8854.092913] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8854.345131] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8864.360439] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8879.693903] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8889.080808] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8889.278501] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8895.075326] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8895.287459] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8900.578409] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8900.579267] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8900.584733] LustreError: Skipped 1 previous similar message [ 8900.587460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8900.603182] Lustre: Skipped 3 previous similar messages [ 8900.607435] Lustre: Skipped 7 previous similar messages [ 8903.911366] Lustre: server umount lustre-OST0000 complete [ 8915.943113] LustreError: 218913:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8916.106669] Lustre: server umount lustre-MDT0000 complete [ 8921.487554] LustreError: 220870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787164793 with bad export cookie 8638653419389767018 [ 8921.492497] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8921.496204] LustreError: 220870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8922.093065] Lustre: server umount lustre-MDT0001 complete [ 8941.631624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8949.703965] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8959.154018] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8964.626559] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 8975.315698] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8983.338695] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8992.613537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8999.373558] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 9008.324279] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9024.993978] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9031.618532] Lustre: server umount lustre-OST0000 complete [ 9041.380142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9041.390897] LustreError: Skipped 2 previous similar messages [ 9041.407864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9041.423524] Lustre: Skipped 4 previous similar messages [ 9041.431304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9041.438540] Lustre: Skipped 6 previous similar messages [ 9043.782890] Lustre: server umount lustre-MDT0000 complete [ 9048.383051] LustreError: 223075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787164920 with bad export cookie 8638653419389767900 [ 9048.390753] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9048.395519] LustreError: 223075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9048.900750] Lustre: server umount lustre-MDT0001 complete [ 9150.523360] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 9153.573378] Key type lgssc unregistered [ 9153.796366] LNet: 226779:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9153.804294] LNetError: 226779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9153.819746] LNet: Removed LNI 192.168.202.115@tcp [ 9154.797330] Key type .llcrypt unregistered [ 9154.801837] Key type ._llcrypt unregistered [ 9184.942312] Key type ._llcrypt registered [ 9184.944414] Key type .llcrypt registered [ 9185.045887] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 9203.282568] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 9204.255871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9204.414861] alg: No test for adler32 (adler32-zlib) [ 9205.522669] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 9205.730351] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 9207.376235] Key type lgssc registered [ 9208.436471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9247.311447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9247.333328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9248.587938] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9248.606622] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9248.696443] Lustre: lustre-MDT0000: new disk, initializing [ 9248.839122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9248.910536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9254.411263] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 9266.346540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9266.493846] Lustre: 230134:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9266.568531] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9266.576137] Lustre: Skipped 1 previous similar message [ 9266.700805] Lustre: lustre-MDT0001: new disk, initializing [ 9266.826270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9266.906474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9266.935486] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9271.450568] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 9281.446852] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9288.486270] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9296.014109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9296.386990] Lustre: lustre-OST0000: new disk, initializing [ 9296.393972] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9296.407292] Lustre: 232104:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9296.558759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9302.085664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9302.094485] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9302.297234] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9304.549373] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 9314.177726] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9319.863141] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9320.036910] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9326.118569] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9326.451654] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9333.232551] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9333.234202] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9333.236524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9333.259153] Lustre: Skipped 1 previous similar message [ 9333.276976] LustreError: Skipped 1 previous similar message [ 9335.003810] Lustre: server umount lustre-OST0000 complete [ 9343.458951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9343.473113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9343.481024] Lustre: Skipped 1 previous similar message [ 9345.264389] Lustre: server umount lustre-MDT0000 complete [ 9349.122585] LustreError: 232103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787165221 with bad export cookie 5326834337240604139 [ 9349.129152] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9349.136786] LustreError: 232103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9349.503409] Lustre: server umount lustre-MDT0001 complete [ 9372.340602] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 9383.288778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9383.931710] LustreError: 235037:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9383.955464] LustreError: 235037:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9384.034717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9389.027046] LustreError: 235038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9389.050304] LustreError: 235038:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9389.450882] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 9394.161056] LustreError: 235037:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9399.266848] LustreError: 235038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9400.582463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9400.957952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9406.442740] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 9410.500954] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9417.118554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9417.679100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9425.831310] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 9434.138408] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9434.263473] Lustre: lustre-OST0001: new disk, initializing [ 9434.266749] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9434.272597] Lustre: 237347:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9434.346786] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9440.858365] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 9441.409286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9441.426853] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9441.548471] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9451.794168] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9459.840282] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9474.347604] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 14:49:05 (1787165345) [ 9477.604900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9477.609151] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9477.611357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9477.621426] Lustre: Skipped 3 previous similar messages [ 9482.728810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9482.755359] Lustre: Skipped 6 previous similar messages [ 9483.371792] Lustre: server umount lustre-MDT0000 complete [ 9487.297593] LustreError: 235018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787165359 with bad export cookie 5326834337240605021 [ 9487.309909] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9487.320098] LustreError: 235018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9487.788324] Lustre: server umount lustre-MDT0001 complete [ 9502.278688] Lustre: server umount lustre-OST0000 complete [ 9504.230368] Lustre: 228359:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787165360/real 1787165360] req@ffff9a0fc27e5f80 x1873978408944000/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787165376 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9504.295243] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9504.319310] Lustre: Skipped 1 previous similar message [ 9507.500471] Lustre: server umount lustre-OST0001 complete [ 9516.322830] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 9528.607520] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 9567.597128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9567.886365] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9567.912099] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9567.973848] Lustre: lustre-MDT0000: new disk, initializing [ 9568.039687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9568.055668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9572.628756] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 9582.656326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9582.766785] Lustre: 241294:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9582.824322] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9582.828968] Lustre: Skipped 1 previous similar message [ 9582.930376] Lustre: lustre-MDT0001: new disk, initializing [ 9583.127417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9583.145693] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9587.276808] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 9597.065306] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9604.219445] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9611.512252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9611.762293] Lustre: lustre-OST0000: new disk, initializing [ 9611.768793] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9611.773360] Lustre: 243263:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9613.907189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9613.917259] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9614.027511] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9618.857357] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 9629.497895] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9634.275774] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9634.559900] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9639.563317] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9639.806906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9645.031826] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9645.033345] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9645.049883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9645.073361] Lustre: Skipped 1 previous similar message [ 9655.266335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9655.272484] Lustre: Skipped 3 previous similar messages [ 9656.289970] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9656.441892] Lustre: server umount lustre-OST0000 complete [ 9660.389437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9660.405199] Lustre: Skipped 1 previous similar message [ 9666.656608] Lustre: server umount lustre-MDT0000 complete [ 9670.534527] LustreError: 241287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787165542 with bad export cookie 5326834337240607156 [ 9670.545628] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9670.560464] LustreError: 241287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9670.956785] Lustre: server umount lustre-MDT0001 complete [ 9679.780817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9680.168974] LustreError: 245480:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9680.273344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9680.278575] Lustre: Skipped 2 previous similar messages [ 9685.820517] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 9690.597408] LustreError: 245481:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9690.622849] LustreError: 245481:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9696.419664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9704.084203] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 9714.159292] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9721.967646] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9728.898085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9737.836812] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [ 9747.196239] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9894.370287] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9894.380319] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9894.389521] LustreError: Skipped 1 previous similar message [ 9894.394850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9894.402060] Lustre: Skipped 2 previous similar messages [ 9894.422318] Lustre: Skipped 5 previous similar messages [ 9908.194428] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9908.557418] Lustre: server umount lustre-OST0000 complete [ 9927.136164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9927.543235] Lustre: server umount lustre-MDT0000 complete [ 9930.212591] LustreError: 245476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9930.226369] LustreError: 245476:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9931.363212] LustreError: 245461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787165803 with bad export cookie 5326834337240608038 [ 9931.366429] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9931.378817] LustreError: 245461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9931.794064] Lustre: server umount lustre-MDT0001 complete [10019.145609] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [10022.894164] Key type lgssc unregistered [10023.336267] LNet: 249091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10023.344463] LNetError: 249091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10023.370309] LNet: Removed LNI 192.168.202.115@tcp [10024.265207] Key type .llcrypt unregistered [10024.267572] Key type ._llcrypt unregistered [10054.266577] Key type ._llcrypt registered [10054.269047] Key type .llcrypt registered [10054.485989] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [10076.267950] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [10077.910806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10078.003949] alg: No test for adler32 (adler32-zlib) [10079.253570] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [10079.531415] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [10081.245272] Key type lgssc registered [10083.353049] Lustre: Echo OBD driver; http://www.lustre.org/ [10141.964649] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [10154.154719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10154.176322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10155.537376] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10155.588530] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10155.695752] Lustre: lustre-MDT0000: new disk, initializing [10155.874504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10155.908708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10160.000782] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10171.325680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10171.456944] Lustre: 253181:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10171.519758] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10171.523404] Lustre: Skipped 1 previous similar message [10171.598816] Lustre: lustre-MDT0001: new disk, initializing [10171.711207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10171.753845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10171.765173] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10176.592888] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10182.366485] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10190.532303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10190.889151] Lustre: lustre-OST0000: new disk, initializing [10190.892353] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10190.903957] Lustre: 254826:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10191.054154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10198.185034] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10199.670550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10199.694914] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10199.849607] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10210.511539] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10210.796780] Lustre: lustre-OST0001: new disk, initializing [10210.801064] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10210.809201] Lustre: 255694:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10210.903261] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10217.483231] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10220.730327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10220.740906] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10220.873608] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10230.228467] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10238.055163] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10251.749690] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10251.762113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10251.781520] Lustre: Skipped 2 previous similar messages [10251.805803] Lustre: Skipped 3 previous similar messages [10256.867407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10256.881562] Lustre: Skipped 2 previous similar messages [10257.324202] Lustre: server umount lustre-MDT0000 complete [10261.467514] LustreError: 253975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787166133 with bad export cookie 9420466921570380534 [10261.481181] LustreError: 253975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10261.484727] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10262.088932] Lustre: server umount lustre-MDT0001 complete [10277.282923] Lustre: server umount lustre-OST0000 complete [10278.374761] Lustre: 250671:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787166134/real 1787166134] req@ffff9a0fd073b800 x1873979324313088/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787166150 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10278.406420] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10278.427550] Lustre: Skipped 1 previous similar message [10281.875043] Lustre: server umount lustre-OST0001 complete [10297.476336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10309.388604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10319.951249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10330.980820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10353.882647] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [10367.617189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10367.649577] Lustre: lustre-MDT0000: reset Object Index mappings [10368.392488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10373.609788] LustreError: 260705:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10373.631601] LustreError: 260705:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10374.224691] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10377.722397] LustreError: 260706:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10382.831029] LustreError: 260705:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10384.511126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10384.525684] Lustre: lustre-MDT0001: reset Object Index mappings [10384.895767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10389.696899] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10393.797730] Lustre: 261849:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10402.428751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10403.005256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10410.156690] LustreError: 262201:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10410.505311] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10413.279719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10418.677029] LustreError: 262533:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10418.707693] LustreError: 262533:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10419.426482] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10424.889383] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10426.449681] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10434.830902] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10438.615559] Lustre: 263718:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10445.299793] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10445.305600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10445.331459] Lustre: Skipped 2 previous similar messages [10445.349181] Lustre: Skipped 3 previous similar messages [10446.328501] Lustre: server umount lustre-MDT0000 complete [10450.413701] LustreError: 262224:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10450.441706] LustreError: 262224:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10450.908698] LustreError: 260686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787166323 with bad export cookie 9420466921570387324 [10450.925121] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10451.463707] Lustre: server umount lustre-MDT0001 complete [10465.865975] Lustre: server umount lustre-OST0000 complete [10480.599099] Lustre: server umount lustre-OST0001 complete [10503.682030] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 15:06:14 (1787166374) [10524.202382] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [10534.921170] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [10585.789218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10586.002812] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10586.027656] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10586.086414] Lustre: lustre-MDT0000: new disk, initializing [10586.144441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10586.147456] Lustre: Skipped 1 previous similar message [10586.170910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10590.949645] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10603.499462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10603.639526] Lustre: 267981:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10603.695902] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10603.699324] Lustre: Skipped 1 previous similar message [10603.814952] Lustre: lustre-MDT0001: new disk, initializing [10603.962025] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10603.988195] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10608.513481] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10618.033980] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10624.420070] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10631.756397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10632.100645] Lustre: lustre-OST0000: new disk, initializing [10632.103893] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10632.111581] Lustre: 269948:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10633.659404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10633.670425] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10633.777697] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10638.346147] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10647.988342] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10654.068661] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10654.274673] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10659.483500] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10659.703144] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10664.420330] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10664.442564] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10664.469375] Lustre: Skipped 1 previous similar message [10664.482063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10667.978885] Lustre: server umount lustre-OST0000 complete [10674.682692] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10674.727778] Lustre: Skipped 1 previous similar message [10674.743494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10674.753553] Lustre: Skipped 2 previous similar messages [10678.470247] Lustre: server umount lustre-MDT0000 complete [10679.793684] LustreError: 268857:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10683.213087] LustreError: 267971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787166555 with bad export cookie 9420466921570389417 [10683.226135] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10683.248554] LustreError: 267971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10683.743157] Lustre: server umount lustre-MDT0001 complete [10704.872728] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [10716.133993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10716.563988] LustreError: 272886:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10716.588188] LustreError: 272886:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10716.665972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10716.671291] Lustre: Skipped 2 previous similar messages [10720.855821] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10728.599193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10733.103775] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10736.393846] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10744.455943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10751.264341] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10761.598818] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10761.754867] Lustre: lustre-OST0001: new disk, initializing [10761.759636] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10761.766509] Lustre: 275196:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10763.193198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10763.208790] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10763.341123] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10769.577945] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10781.355845] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10785.483765] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10791.675677] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10791.748302] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10799.077745] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10799.087510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10799.098993] Lustre: Skipped 4 previous similar messages [10799.123343] Lustre: Skipped 2 previous similar messages [10805.274991] Lustre: server umount lustre-MDT0000 complete [10809.118910] LustreError: 272866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787166681 with bad export cookie 9420466921570390299 [10809.122926] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10809.132148] LustreError: 272866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10809.360895] Lustre: server umount lustre-MDT0001 complete [10823.656238] Lustre: server umount lustre-OST0000 complete [10824.672793] Lustre: 250673:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787166681/real 1787166681] req@ffff9a0fd09c8000 x1873979324472320/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787166697 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10824.703137] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10828.500318] Lustre: server umount lustre-OST0001 complete [10850.021937] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [10864.350176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10864.723288] LustreError: 278249:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10864.738412] LustreError: 278249:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10864.820101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10864.830505] Lustre: Skipped 3 previous similar messages [10871.017588] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10881.361692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10886.716590] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10890.001231] Lustre: 279389:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10897.912166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10905.624652] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10909.630251] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10914.970741] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10920.433395] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10922.235884] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [10930.620755] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10934.671489] Lustre: 281267:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10940.341777] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10940.372946] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10951.924821] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 15:13:43 (1787166823) [10969.057413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10969.067830] LustreError: Skipped 1 previous similar message [10969.070806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10969.084371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10969.092717] Lustre: Skipped 5 previous similar messages [10982.368158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10982.519729] Lustre: server umount lustre-MDT0000 complete [10987.199253] LustreError: 278230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787166859 with bad export cookie 9420466921570396627 [10987.206748] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10987.218906] LustreError: 278230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10993.994991] Lustre: server umount lustre-MDT0001 complete [11004.217471] Lustre: server umount lustre-OST0000 complete [11008.135179] Lustre: server umount lustre-OST0001 complete [11015.342534] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [11026.896958] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [11076.013511] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [11086.188083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11086.408650] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11086.438250] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11086.510329] Lustre: lustre-MDT0000: new disk, initializing [11086.588332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11091.162790] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11100.738361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11100.893790] Lustre: 285517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11101.178447] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11106.500308] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11118.950048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11119.240304] Lustre: 287150:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11121.093574] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11127.198641] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11138.958267] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11139.191219] Lustre: 288020:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11139.344581] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11139.348882] Lustre: Skipped 6 previous similar messages [11140.437768] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11140.447597] Lustre: Skipped 1 previous similar message [11140.599932] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11147.340426] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11158.188862] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11161.673497] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11161.679534] Lustre: Skipped 1 previous similar message [11177.564393] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 15:17:28 (1787167048) [11181.544246] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11181.548640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11181.558257] Lustre: Skipped 7 previous similar messages [11181.567133] Lustre: Skipped 17 previous similar messages [11187.007765] Lustre: server umount lustre-MDT0000 complete [11190.891069] LustreError: 285510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787167063 with bad export cookie 9420466921570404495 [11190.898238] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11190.903471] LustreError: 285510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11191.262623] Lustre: server umount lustre-MDT0001 complete [11206.130830] Lustre: server umount lustre-OST0000 complete [11207.905944] Lustre: 250672:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787167064/real 1787167064] req@ffff9a10c9b18380 x1873979324632576/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787167080 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11211.398528] Lustre: server umount lustre-OST0001 complete [11223.821754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11224.458563] LustreError: 290267:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11224.480212] LustreError: 290267:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [11229.276588] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11241.129986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11247.060566] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11255.649360] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11262.409819] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11270.794843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11278.284456] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11287.220846] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11318.098335] loop1: detected capacity change from 0 to 204800000 [11318.320413] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11318.333565] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11318.348719] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11323.812748] loop1: detected capacity change from 0 to 204800000 [11329.637539] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11329.719455] LustreError: 290256:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs15879: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11329.733587] LustreError: 290256:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs15879 [11329.738478] LustreError: 290256:0:(mgs_handler.c:517:mgs_target_reg()) fs15879-MDT0001: Error checking target index: rc = -13 [11329.747584] LustreError: fs15879-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11329.761539] LustreError: 294010:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11329.778394] LustreError: 294010:0:(tgt_mount.c:2129:server_put_super()) no obd fs15879-MDT0001 [11329.783707] LustreError: 294010:0:(tgt_mount.c:116:server_deregister_mount()) fs15879-MDT0001 not registered [11329.800044] Lustre: server umount fs15879-MDT0001 complete [11329.802526] LustreError: 294010:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [11339.116077] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11339.397750] Lustre: Setting parameter fs15879-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15879-MDT0001 [11339.453855] Lustre: srv-fs15879-MDT0001: No data found on store. Initialize space. [11339.457996] Lustre: Skipped 4 previous similar messages [11339.563376] Lustre: fs15879-MDT0001: new disk, initializing [11339.567204] Lustre: Skipped 3 previous similar messages [11339.777064] Lustre: srv-fs15879-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11339.785323] Lustre: 294477:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs15879-MDT0001-mdtlov: can't create new llog: rc = -115 [11339.800440] LustreError: 294477:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs15879-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11344.786371] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11351.255461] Lustre: server umount fs15879-MDT0001 complete [11365.879590] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11366.355255] loop1: detected capacity change from 0 to 204800000 [11370.985865] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11371.342344] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11371.733477] Lustre: 296046:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs15879-MDT0001-mdtlov: can't create new llog: rc = -115 [11371.757929] LustreError: 296046:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs15879-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11377.494681] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11382.461986] Lustre: server umount fs15879-MDT0001 complete [11395.553963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11395.569219] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11395.582283] Lustre: Skipped 2 previous similar messages [11395.586912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11395.591153] Lustre: Skipped 4 previous similar messages [11398.501582] Lustre: server umount lustre-MDT0000 complete [11407.361451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11407.498172] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11412.798804] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11412.980168] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11421.288741] loop1: detected capacity change from 0 to 204800000 [11427.517261] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11427.760592] Lustre: 298493:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs15879-MDT0001-mdtlov: can't create new llog: rc = -115 [11427.771051] LustreError: 298493:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs15879-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11433.101693] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11440.300101] Lustre: server umount fs15879-MDT0001 complete [11464.162971] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11470.243273] Lustre: server umount lustre-OST0000 complete [11480.330790] Lustre: server umount lustre-MDT0000 complete [11484.055606] LustreError: 290249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787167356 with bad export cookie 9420466921570407176 [11484.057323] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11484.067040] LustreError: 290249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11484.464133] Lustre: server umount lustre-MDT0001 complete [11498.407338] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [11501.274787] Key type lgssc unregistered [11501.617914] LNet: 300838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11501.629353] LNetError: 300838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11501.653722] LNet: Removed LNI 192.168.202.115@tcp [11502.814228] Key type .llcrypt unregistered [11502.818736] Key type ._llcrypt unregistered [11524.358467] Key type ._llcrypt registered [11524.360942] Key type .llcrypt registered [11524.473426] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [11525.552430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11525.719399] alg: No test for adler32 (adler32-zlib) [11526.802335] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [11527.029932] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [11528.728436] Key type lgssc registered [11530.221737] Lustre: Echo OBD driver; http://www.lustre.org/ [11539.168414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11539.191974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11541.114844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11547.094743] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11551.221632] LustreError: 302108:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11551.237515] LustreError: 302108:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11556.324178] LustreError: 302109:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11557.214169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11557.850292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11562.908507] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11572.685200] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11582.698908] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11592.501913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11593.022858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11601.890748] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11603.277161] LustreError: 303939:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11608.551315] LustreError: 303941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11611.277956] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11614.076206] LustreError: 303941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11614.103139] LustreError: 303941:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11624.316275] LustreError: 303939:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11624.341516] LustreError: 303939:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11629.406878] loop1: detected capacity change from 0 to 204800000 [11629.654622] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11629.679081] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11629.693091] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11634.550849] loop1: detected capacity change from 0 to 204800000 [11640.373790] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11640.568523] Lustre: fs21655-OST0000: new disk, initializing [11640.571551] Lustre: srv-fs21655-OST0000: No data found on store. Initialize space. [11640.577839] Lustre: 305846:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11640.665786] Lustre: fs21655-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11642.342174] LustreError: 303939:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11642.373214] LustreError: 303939:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [11648.731397] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11658.283135] Lustre: server umount fs21655-OST0000 complete [11671.849184] loop1: detected capacity change from 0 to 204800000 [11675.506520] LustreError: 303941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11675.536693] LustreError: 303941:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [11678.418367] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11678.744073] Lustre: fs21655-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11683.325761] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11685.855372] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11688.436876] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11695.932406] loop2: detected capacity change from 0 to 204800000 [11696.145124] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11696.163464] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11696.185450] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11702.758357] loop2: detected capacity change from 0 to 204800000 [11710.068318] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11710.230490] LustreError: fs21655-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11710.260745] LustreError: fs21655-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11710.270582] LustreError: 308890:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11710.281210] LustreError: 308890:0:(tgt_mount.c:2129:server_put_super()) no obd fs21655-OST0001 [11710.303292] LustreError: 308890:0:(tgt_mount.c:116:server_deregister_mount()) fs21655-OST0001 not registered [11710.383214] Lustre: server umount fs21655-OST0001 complete [11710.402217] LustreError: 308890:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [11726.143320] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11726.601400] Lustre: fs21655-OST0001: new disk, initializing [11726.603919] Lustre: srv-fs21655-OST0001: No data found on store. Initialize space. [11726.611686] Lustre: 309441:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11726.684053] Lustre: fs21655-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11733.260280] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11739.623740] LustreError: 303940:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11739.653475] LustreError: 303940:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 37 previous similar messages [11743.957418] Lustre: server umount fs21655-OST0000 complete [11755.513486] Lustre: server umount fs21655-OST0001 complete [11771.940963] loop1: detected capacity change from 0 to 204800000 [11780.390548] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11780.747466] Lustre: fs21655-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11789.066519] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11797.996753] loop2: detected capacity change from 0 to 204800000 [11804.310479] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11812.749462] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [11823.753913] Lustre: server umount fs21655-OST0000 complete [11833.342902] Lustre: server umount fs21655-OST0001 complete [11836.932588] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11842.029091] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11852.260124] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11852.307488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11852.308497] Lustre: Skipped 1 previous similar message [11852.331907] Lustre: Skipped 1 previous similar message [11857.181294] Lustre: server umount lustre-OST0000 complete [11862.497228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11862.504488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11862.517836] Lustre: Skipped 1 previous similar message [11862.536811] Lustre: Skipped 1 previous similar message [11867.620512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11867.633087] Lustre: Skipped 1 previous similar message [11871.718028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11871.732550] Lustre: Skipped 1 previous similar message [11876.324096] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11876.615583] Lustre: server umount lustre-MDT0000 complete [11876.833717] LustreError: 303600:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11876.859356] LustreError: 303600:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 66 previous similar messages [11881.012732] LustreError: 302090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787167753 with bad export cookie 7517493656305538294 [11881.014915] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11881.028412] LustreError: 302090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11881.602976] Lustre: server umount lustre-MDT0001 complete [11893.281028] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [11895.916212] Key type lgssc unregistered [11896.249985] LNet: 315371:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11896.256993] LNetError: 315371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11896.280240] LNet: Removed LNI 192.168.202.115@tcp [11897.085485] Key type .llcrypt unregistered [11897.088404] Key type ._llcrypt unregistered [11936.868458] Key type ._llcrypt registered [11936.870013] Key type .llcrypt registered [11937.012489] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [11956.794384] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [11958.570259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11958.697039] alg: No test for adler32 (adler32-zlib) [11959.873759] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [11960.148064] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [11961.841389] Key type lgssc registered [11964.067580] Lustre: Echo OBD driver; http://www.lustre.org/ [12010.832973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12010.886663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12012.285324] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12012.333667] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12012.560377] Lustre: lustre-MDT0000: new disk, initializing [12012.842328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12012.894068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12018.261516] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [12031.060769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12031.184752] Lustre: 319241:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12031.235650] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12031.239157] Lustre: Skipped 1 previous similar message [12031.308879] Lustre: lustre-MDT0001: new disk, initializing [12031.371933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12031.431977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12031.447596] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12036.239881] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [12047.724100] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12055.579573] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12063.476672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12063.870097] Lustre: lustre-OST0000: new disk, initializing [12063.877796] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12063.898359] Lustre: 321212:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12064.090215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12070.495436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12070.519954] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12070.650786] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12071.561765] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [12082.891602] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12085.379233] Lustre: Mounted lustre-client [12086.575590] Lustre: Unmounted lustre-client [12101.092231] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12101.113420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12101.124475] Lustre: Skipped 1 previous similar message [12103.029707] Lustre: server umount lustre-OST0000 complete [12108.257823] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12108.266248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12108.277372] Lustre: Skipped 1 previous similar message [12108.287441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12111.335620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12111.339928] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12114.053928] Lustre: server umount lustre-MDT0000 complete [12118.895360] LustreError: 319234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787167991 with bad export cookie 5688547222536091264 [12118.900900] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12118.908705] LustreError: 319234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12119.593679] Lustre: server umount lustre-MDT0001 complete [12138.460346] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [12142.345343] Key type lgssc unregistered [12142.613235] LNet: 323389:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12142.617436] LNetError: 323389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12215.335766] LNet: Removed LNI 192.168.202.115@tcp [12216.526903] Key type .llcrypt unregistered [12216.530385] Key type ._llcrypt unregistered [12239.204874] Key type ._llcrypt registered [12239.210375] Key type .llcrypt registered [12239.435391] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [12267.294180] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [12268.188160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12268.354316] alg: No test for adler32 (adler32-zlib) [12269.714290] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [12270.003345] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [12271.824162] Key type lgssc registered [12273.360754] Lustre: Echo OBD driver; http://www.lustre.org/ [12310.502701] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 15:36:21 (1787168181) [12318.977886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12319.026662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12320.551283] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12320.613785] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12320.819771] Lustre: lustre-MDT0000: new disk, initializing [12321.104563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12321.140853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12327.655159] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [12339.405239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12339.569507] Lustre: 326613:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12339.661603] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12339.667030] Lustre: Skipped 1 previous similar message [12339.732149] Lustre: lustre-MDT0001: new disk, initializing [12339.818649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12339.886216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12339.899752] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12345.269197] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [12358.158632] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12365.207470] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12373.657429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12373.897318] Lustre: lustre-OST0000: new disk, initializing [12373.901837] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12373.909212] Lustre: 328581:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12374.033467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12381.236647] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [12381.264601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12381.273142] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12381.376520] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12394.137961] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12412.389987] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12412.402981] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12412.414418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12412.422914] Lustre: Skipped 1 previous similar message [12417.040126] Lustre: server umount lustre-OST0000 complete [12426.728908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12426.731842] LustreError: Skipped 1 previous similar message [12426.748578] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12426.770087] Lustre: Skipped 1 previous similar message [12426.772333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12427.779719] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12427.808303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12427.817216] Lustre: Skipped 1 previous similar message [12430.274183] Lustre: server umount lustre-MDT0000 complete [12435.020380] LustreError: 326604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787168307 with bad export cookie 17204667648590047257 [12435.021941] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12435.026193] LustreError: 326604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12435.455069] Lustre: server umount lustre-MDT0001 complete [12452.802528] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [12457.245667] Key type lgssc unregistered [12457.658901] LNet: 330513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12457.674111] LNetError: 330513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12458.728331] LNet: Removed LNI 192.168.202.115@tcp [12459.746738] Key type .llcrypt unregistered [12459.749342] Key type ._llcrypt unregistered [12485.469977] Key type ._llcrypt registered [12485.472531] Key type .llcrypt registered [12485.560532] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [12486.438868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12486.538772] alg: No test for adler32 (adler32-zlib) [12487.822841] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [12488.005329] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [12489.700363] Key type lgssc registered [12491.303122] Lustre: Echo OBD driver; http://www.lustre.org/ [12500.149976] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12500.170509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12501.723541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12506.740334] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [12512.228302] LustreError: 331786:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12512.249546] LustreError: 331786:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12516.082439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12516.449831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12521.698776] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [12531.022158] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12538.707693] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12548.090909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12548.511098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12557.252993] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [12569.049586] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12584.941905] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12586.491659] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12591.504349] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.15@tcp (stopping) [12593.127192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12593.144580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12593.171298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12593.646421] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12593.665779] Lustre: Skipped 1 previous similar message [12596.600143] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.15@tcp (stopping) [12596.615131] Lustre: Skipped 2 previous similar messages [12597.675808] Lustre: server umount lustre-MDT0000 complete [12598.764507] LustreError: 331785:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12598.797858] LustreError: 331785:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12601.715058] LustreError: 333559:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12602.802399] LustreError: 331768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787168475 with bad export cookie 6601791132532221053 [12602.826865] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12603.894344] LustreError: 331782:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12603.896793] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12603.952761] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12607.906409] LustreError: 331780:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12607.936626] LustreError: 331780:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12611.956728] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.15@tcp (stopping) [12611.968175] Lustre: Skipped 2 previous similar messages [12617.096598] LustreError: 334707:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12617.126912] LustreError: 334707:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12617.184135] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12617.880781] Lustre: server umount lustre-MDT0001 complete [12623.092256] Lustre: server umount lustre-OST0000 complete [12633.810147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12639.599430] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [12642.869785] Lustre: 335974:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.115@tcp [12645.229334] Lustre: server umount MGS complete [12655.187201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12655.581533] LustreError: 336545:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12655.674062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12661.259804] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [12671.399157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12671.851762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12677.071540] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [12687.439040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12687.846444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12694.462938] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [12703.206983] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12705.539926] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12713.285057] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12715.126206] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12721.128613] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [12721.137756] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12721.152359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12721.158797] Lustre: Skipped 2 previous similar messages [12723.050811] Lustre: server umount lustre-OST0000 complete [12728.289133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12728.304906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12728.323888] Lustre: Skipped 1 previous similar message [12734.556758] Lustre: server umount lustre-MDT0000 complete [12736.994898] LustreError: 336542:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12737.020622] LustreError: 336542:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12739.381980] LustreError: 337281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787168611 with bad export cookie 6601791132532221900 [12739.389934] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12739.394965] LustreError: 337281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [12739.936632] Lustre: server umount lustre-MDT0001 complete [12758.546385] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [12762.015694] Key type lgssc unregistered [12762.322379] LNet: 340008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12762.334198] LNetError: 340008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12762.371620] LNet: Removed LNI 192.168.202.115@tcp [12764.027984] Key type .llcrypt unregistered [12764.030100] Key type ._llcrypt unregistered [12794.229889] Key type ._llcrypt registered [12794.232709] Key type .llcrypt registered [12794.336758] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [12811.381697] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [12812.591453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12812.785285] alg: No test for adler32 (adler32-zlib) [12814.225386] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [12814.726571] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [12816.504515] Key type lgssc registered [12818.050879] Lustre: Echo OBD driver; http://www.lustre.org/ [12858.391480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12858.440566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12859.671313] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12859.718457] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12859.858168] Lustre: lustre-MDT0000: new disk, initializing [12859.968820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12860.000913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12865.196599] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [12877.567901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12877.642757] Lustre: 343173:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12877.685222] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12877.689367] Lustre: Skipped 1 previous similar message [12877.755604] Lustre: lustre-MDT0001: new disk, initializing [12877.886406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12877.940173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12877.950865] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12883.014759] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [12894.476965] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12900.852543] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12907.235307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12907.675478] Lustre: lustre-OST0000: new disk, initializing [12907.681029] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12907.686623] Lustre: 345142:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12907.818816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12914.390996] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [12917.321812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12917.329946] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12917.454118] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12924.302660] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12929.615592] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12929.815700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12934.013708] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12934.290485] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12938.220603] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12938.225435] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12938.239327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12938.247855] Lustre: Skipped 1 previous similar message [12938.263433] Lustre: Skipped 1 previous similar message [12942.301766] Lustre: server umount lustre-OST0000 complete [12948.474940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12948.477607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12948.481681] Lustre: Skipped 1 previous similar message [12952.561283] Lustre: server umount lustre-MDT0000 complete [12956.874283] LustreError: 343166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787168829 with bad export cookie 12884366249191970491 [12956.880890] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12956.888473] LustreError: 343166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12957.406264] Lustre: server umount lustre-MDT0001 complete [12971.467125] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 15:47:21 (1787168841) [12997.958717] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [13011.058090] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [13052.445649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13052.691064] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13052.750758] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13052.838571] Lustre: lustre-MDT0000: new disk, initializing [13052.977554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13053.003477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13058.386840] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [13070.783437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13070.921488] Lustre: 349904:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13070.979187] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13070.985066] Lustre: Skipped 1 previous similar message [13071.069407] Lustre: lustre-MDT0001: new disk, initializing [13071.149485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13071.214334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13071.224544] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13076.802657] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [13088.200551] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13096.558699] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13107.285165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13107.917695] Lustre: lustre-OST0000: new disk, initializing [13107.936657] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13107.958655] Lustre: 351877:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13108.151854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13117.137375] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [13117.180893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13117.200523] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13117.333342] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13130.073723] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13173.736356] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13173.738207] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13173.738972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13173.738977] Lustre: Skipped 1 previous similar message [13173.764880] LustreError: Skipped 2 previous similar messages [13173.801216] Lustre: Skipped 1 previous similar message [13178.953829] Lustre: server umount lustre-OST0000 complete [13183.969843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13183.978409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13183.989770] Lustre: Skipped 1 previous similar message [13184.004108] Lustre: Skipped 2 previous similar messages [13189.093616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13189.105440] Lustre: Skipped 1 previous similar message [13190.013517] Lustre: server umount lustre-MDT0000 complete [13194.217794] LustreError: 349916:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13194.242066] LustreError: 349916:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13194.322440] LustreError: 349896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787169066 with bad export cookie 12884366249191971373 [13194.327360] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13194.337539] LustreError: 349896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13195.141727] Lustre: server umount lustre-MDT0001 complete [13213.021603] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [13216.864342] Key type lgssc unregistered [13217.179877] LNet: 354073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13217.193970] LNetError: 354073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13217.254191] LNet: Removed LNI 192.168.202.115@tcp [13218.704188] Key type .llcrypt unregistered [13218.710766] Key type ._llcrypt unregistered [13341.005398] Key type ._llcrypt registered [13341.008158] Key type .llcrypt registered [13341.136745] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [13341.526901] Key type .llcrypt unregistered [13341.535529] Key type ._llcrypt unregistered [13441.431711] Key type ._llcrypt registered [13441.435586] Key type .llcrypt registered [13441.558853] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [13442.005407] Key type .llcrypt unregistered [13442.007251] Key type ._llcrypt unregistered [13483.765442] Key type ._llcrypt registered [13483.771466] Key type .llcrypt registered [13483.899159] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [13484.192391] Key type .llcrypt unregistered [13484.196523] Key type ._llcrypt unregistered [13528.517183] Key type ._llcrypt registered [13528.518889] Key type .llcrypt registered [13528.722586] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [13529.156611] Key type .llcrypt unregistered [13529.162292] Key type ._llcrypt unregistered [13626.002642] Key type ._llcrypt registered [13626.004846] Key type .llcrypt registered [13626.133452] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [13626.498840] Key type .llcrypt unregistered [13626.501450] Key type ._llcrypt unregistered [13664.043912] Key type ._llcrypt registered [13664.045400] Key type .llcrypt registered [13664.114971] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [13664.433590] Key type .llcrypt unregistered [13664.436388] Key type ._llcrypt unregistered [13702.362934] Key type ._llcrypt registered [13702.367805] Key type .llcrypt registered [13702.562094] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [13703.141362] Key type .llcrypt unregistered [13703.150113] Key type ._llcrypt unregistered [13739.894101] Key type ._llcrypt registered [13739.897567] Key type .llcrypt registered [13740.021781] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [13740.298502] Key type .llcrypt unregistered [13740.301704] Key type ._llcrypt unregistered [13776.712899] Key type ._llcrypt registered [13776.719144] Key type .llcrypt registered [13776.860241] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [13777.209415] Key type .llcrypt unregistered [13777.212208] Key type ._llcrypt unregistered [13813.001609] Key type ._llcrypt registered [13813.004396] Key type .llcrypt registered [13813.131277] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [13813.576324] Key type .llcrypt unregistered [13813.581972] Key type ._llcrypt unregistered [13897.557745] Key type ._llcrypt registered [13897.560476] Key type .llcrypt registered [13897.646274] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [13897.851632] Key type .llcrypt unregistered [13897.854466] Key type ._llcrypt unregistered [13933.387660] Key type ._llcrypt registered [13933.389741] Key type .llcrypt registered [13933.559629] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [13933.965227] Key type .llcrypt unregistered [13933.969220] Key type ._llcrypt unregistered [13972.952898] Key type ._llcrypt registered [13972.957919] Key type .llcrypt registered [13973.062771] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [13973.279754] Key type .llcrypt unregistered [13973.284571] Key type ._llcrypt unregistered [14039.592290] Key type ._llcrypt registered [14039.595255] Key type .llcrypt registered [14039.699976] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [14061.475288] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [14063.355693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14063.405066] alg: No test for adler32 (adler32-zlib) [14064.726803] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [14065.085516] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [14066.928255] Key type lgssc registered [14068.711875] Lustre: Echo OBD driver; http://www.lustre.org/ [14119.351071] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14119.378535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14121.463875] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14121.529953] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14121.741284] Lustre: tst250fs-MDT0000: new disk, initializing [14121.960644] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14121.996463] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14129.804886] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [14144.857299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14144.984188] Lustre: 374500:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14145.070425] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14145.080593] Lustre: Skipped 1 previous similar message [14145.226823] Lustre: tst250fs-MDT0001: new disk, initializing [14145.412557] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14145.520154] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14145.540062] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14151.041447] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [14156.597238] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14163.973658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14164.196616] Lustre: tst250fs-OST0000: new disk, initializing [14164.199793] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14164.206595] Lustre: 376131:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14164.338787] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14167.818964] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14167.837389] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14167.990240] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14171.826944] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [14183.803893] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14183.962949] Lustre: tst250fs-OST0001: new disk, initializing [14183.966430] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14183.975759] Lustre: 377001:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14184.074441] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14192.091416] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all [14192.777438] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14192.791978] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14192.906428] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14198.325215] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14200.710595] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [14200.715710] Lustre: Skipped 1 previous similar message [14223.851049] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14223.857447] Lustre: Skipped 1 previous similar message [14223.859236] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14229.028781] Lustre: server umount tst250fs-MDT0000 complete [14233.418038] LustreError: 376130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787170105 with bad export cookie 540813829391581205 [14233.424792] LustreError: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14233.429879] LustreError: 376130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14233.771912] Lustre: server umount tst250fs-MDT0001 complete [14249.547136] Lustre: server umount tst250fs-OST0000 complete [14250.465111] Lustre: 372720:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787170106/real 1787170106] req@ffff9a0fc8ad0700 x1873983503931136/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787170122 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14250.491216] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14250.517139] Lustre: Skipped 2 previous similar messages [14254.396302] Lustre: server umount tst250fs-OST0001 complete [14278.482220] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [14289.750858] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [14324.396459] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 16:09:55 (1787170195) [14326.339225] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14347.230514] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [14364.630296] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [14408.822187] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14110 sec ========= 16:11:19 (1787170279) [14411.048397] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:11:22 (1787170282) === [14415.528609] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:11:26 (1787170286) === [14461.587682] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [14464.898026] Key type lgssc unregistered [14465.344422] LNet: 386172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14465.353500] LNetError: 386172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14465.370817] LNet: Removed LNI 192.168.202.115@tcp [14466.488200] Key type .llcrypt unregistered [14466.489893] Key type ._llcrypt unregistered